Thread Rating:
  • 2 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
ANNOUNCING the Skydoms Character Spreadsheet
#1
ANNOUNCING the Skydoms Character Spreadsheet

I've been toying around with this for a few months now, and it's finally time to unleash it on the rest of you, err, "give back to the community".  Smile

[Before anyone asks: I spoke with Skydoms developer Alex.  He's fine with making this public.]

You see, I'd been wishing for a way to quickly browse through ALL of the Skydoms fighters, compare stats and abilities, etc, without constantly going back and forth between multiple pages on the main site.

I started looking at "web scrapers"; programs that could collect data from multiple web pages into a single file.  I quickly settled on Outwit Hub as the best free solution out there, got surprisingly far with it, then said "screw it" and bought the full version specifically so I could do a proper job with this project (which by now had become a full-fledged hobby).

Fast forward a few months, and by now, the whole process is almost entirely automated.  The Outwit "Skydoms Scraper" I built collects every field from every fighter page and saves them as a plain Excel spreadsheet.  (Another part of Outwit handles downloading all of the fighter images.)  I copy the values from the plain worksheet into my previously-formatted spreadsheet, which also contains macros for (re)loading the images, (re)formatting the cells, and generally making everything look awesome.  Finally, one last macro automatically generates two NON-MACRO copies for publication: one with pictures (6MB), and one without (240KB).  

I can generate a completely new spreadsheet from the website and have both versions available for download within half an hour.  (So, the good news is: I can keep the spreadsheet pretty current when the fighters get updated!  Barring any major web page redesigns forcing me to update my scraper code itself, that is...)



WHAT'S THE PURPOSE OF THE SKYDOMS CHARACTER SPREADSHEET?

1) Viewing all of the character data in one place.
2) Being able to sort on and filter various fields; to group the characters in meaningful ways and learn more about them in relation to each other.

For instance, you can sort on the Max Attack values and quickly see which fighters have the highest attack potential.

Sort by Primary School, Secondary School, and Number to get a neat by-color listing, with the two-color fighters sorted at the end of each primary color -- or use the Filter to remove the blank Secondary School rows, if two-color fighters are all you care about.

Sort (or Filter) Kind1 and Kind2 to find all the Healers.  You get the idea.

NOTE: Set the Filters to ALL again when you're done, but do NOT turn off Filtering altogether!  See below.



SPECIAL INSTRUCTIONS FOR SORTING

This spreadsheet has two header rows.  Excel does not handle sorting multi-header-row sheets very well.  In order to help Excel "figure things out", I have Filtering turned on, using a blank third header row as a separator, containing the filter drop-down menus.

This largely solves the problem.  However, YOU MUST MAKE SURE you have one of the non-header cells selected before telling Excel to Sort -- and then make sure that "my data has headers" is NOT checked.  As long as you do that, it will work fine.  Mostly.

[I've been Googling and hacking at this thing for weeks, trying to find a better way for Excel to cleanly sort a multi-header-row spreadsheet.  The BEST solution is to use macros, but I didn't want to distribute anything requiring macros to work.  If anyone out there can look this over and find a better solution, PLEASE let me know.]

Alternately, if you just want to sort on one column, you can simply use the filter drop-down menu in the third row, for the column in question.

You can also use the filter drop-downs to actually FILTER the rows to just the ones you're looking for.  But when you're done, set the filters individually back to ALL; do NOT turn off Filtering, or Sorting will stop working right.  (*SIGH*)

[If you DO accidentally turn off Filtering, select row 3, click the DATA menu, and click FILTER to turn it back on.]

Oh, and do NOT try to sort on a picture column; it's a waste of time.



FINAL REQUEST:

In order to allow sorting, I have to leave the spreadsheet unlocked.  [Yes, Excel is Just That Dumb.]  That's fine.  It's not exactly MY data in the first place.  But the spreadsheet layout IS my design; I ask that you NOT make changes and re-publish it.

If you think that something ought to be changed or done differently, or simply think that something could be improved -- if you have any suggestions, Excel tips, ANYTHING AT ALL -- please send me a private message here on the forums.  (Try to restrict thread responses to things other people might actually care to read.)

I'll be happy to make changes that seem reasonable, if I'm capable of doing it.  

Hell, I'm not done tweaking this thing.  But if I waited until I was DONE, it would never get published.  Smile

I just hope this is at least vaguely helpful to someone...



DOWNLOAD:

There are two versions available to download: WITH and WITHOUT pictures.  The spreadsheet is perfectly usable without pictures; in BOTH versions, all of the actual information has been translated as text.  [Excel will *NOT* allow me to place icons within the sentence text, as the website does for the various Skill Effect descriptions.]

However, I've found it's really hard for me to keep track of which fighter is which, based on just the names.  Besides, the whole thing is so much PRETTIER with the pictures and icons in place... it's worth a little delay in the loading and sorting, right?

These are Excel 2013 .XLSX files with NO MACROS.

(The reason for the URL shortener is simply to allow me to count downloads and see if anyone is actually interested...)

CLICK HERE to download Skydoms Character Spreadsheet WITH pictures (6MB)

CLICK HERE to download Skydoms Character Spreadsheet WITHOUT pictures (240KB)



Tech Note: TRANSLATIONS

Just in case you're interested in the translations that I have Outwit Scraper doing before scraping each page...

The five round color icons are translated to:
GREEN (War)
PURPLE (Shadow)
RED (Elemental)
YELLOW (Mech)
BLUE (Support)

The five square color icons are translated to:
[GREEN TILE]
[PURPLE TILE]
[RED TILE]
[YELLOW TILE]
[BLUE TILE]

Sword icons are translated to: [ATT]
Shield icons are translated to: [DEF]
Lightning icons are translated to: [ENG]
(Lightning icons appearing prior to the phrase "Active Skills" are ignored.)
Heart icons are translated to: [HP]
Dice icons (generally appearing before the word "skills") are translated to: Random

From time to time, you may notice a misspelling on the fighter pages, and notice that it's corrected in the spreadsheet.  Yeah, as long as Outwit is doing global replacements on each page ANYWAY, I'm catching these as well... Smile

Enjoy!
Berek         Diamond           ID=948.549.375
Seeking friends with Beelzebub captains.
I have Pride, Anubis, Set, Nezha...


Reply
#2
Oh my goodness Berek. This is amazing!!!!

*big hug and a smooch on the cheek*

I've been wanting to browse the fighters for awhile now without the interference of constant ads and page loads. Thank you so much for all your hard work.

*runs off to try it out*
Reply
#3
This character sheet is HUGE. But it pays off, as it's very comprehensive. Now I can review everything I need to know about my fighters!
ded gaem, etermax disband
Lineup (Red): Atahualpa, Grimer, Annlis (6★), Fenris, Salamander (5★)
ID: 997.071.092
Reply
#4
Thank you VERY VERY much! Kisses Love
<I don't have to pretend to be less than I am, to make you appear greater than you are.>
Reply
#5
Thanks to everyone for the support and kind words!

Just wanted to announce that the spreadsheet has been updated with the new character data as of 4/29/2016.

Same links as before, but if you don't want to scroll to the top post:

CLICK HERE to download Skydoms Character Spreadsheet WITH pictures (6MB)

CLICK HERE to download Skydoms Character Spreadsheet WITHOUT pictures (240KB)
Berek         Diamond           ID=948.549.375
Seeking friends with Beelzebub captains.
I have Pride, Anubis, Set, Nezha...


Reply
#6
Give 'em credit.  The devs (and artists) are keeping busy cranking out new fighters -- and keeping me on my toes, watching for updates, and updating my spreadsheet accordingly!

I've now uploaded the latest, from 5/8/2016.  As always, the download links remain the same, but are duplicated here for good measure.

[NOTE: I always release full updates.  If there are any changes in previously-existing characters on the Skydoms Wiki pages, they will show up here.]

[ALSO NOTE: Unnoticed web page formatting changes, and even simple labeling changes, can screw up my web scraper logic.  If anyone ever notices anything horribly wrong in my spreadsheet (incorrect data, missing data that IS present on the wiki, or data in the wrong columns), please message me ASAP and I will try to fix it as soon as possible.  Thank you.]

CLICK HERE to download Skydoms Character Spreadsheet WITH pictures (6MB)

CLICK HERE to download Skydoms Character Spreadsheet WITHOUT pictures (240KB)
Berek         Diamond           ID=948.549.375
Seeking friends with Beelzebub captains.
I have Pride, Anubis, Set, Nezha...


Reply
#7
I have a fully comprehensive spreadsheet I finalized in Access so that its sortable and not changeable. I've shared it to my clan members but never thought to go public. If you want me to teach you how to lock your grid but allow sorting I'd be more than happy too. Or teach you how to publish it to a blog post so that its sortable by each category and - again - unchangeable by viewers.
Reply
#8
[I sent a private message to Jessie (see prev message); never heard back.]


I see we have a few new fighters again.  So the spreadsheet is updated, as of early this morning (5/22/2016).  Same links as always.

CLICK HERE to download Skydoms Character Spreadsheet WITH pictures (6MB)

CLICK HERE to download Skydoms Character Spreadsheet WITHOUT pictures (240KB)
Berek         Diamond           ID=948.549.375
Seeking friends with Beelzebub captains.
I have Pride, Anubis, Set, Nezha...


Reply
#9
YEESH.  Sorry about that.  It took much longer than usual for me to get the spreadsheet updated this time.

It's not the number of new characters -- though there's a bunch.  

It's the fact that the devs went and tweaked the formatting of the fighter pages a little.

*VERY* little, mind you.  But it was enough to massively break my scraper regular expressions, which took some time to re-sort-out.  Plus one of the changes involved adding a whole new picture column to the each fighter page, showing who they evolve FROM -- which translates to TWO new columns for me, plus rejiggering the picture-loading macros.  (Fortunately, I wrote my macros right the first time, and it's no problem to accommodate new or even rearranged columns.)

Anyway, I've got all of my automation is up and running again.  And the latest spreadsheet is ready for download!

[Observant downloaders will notice one new character that the devs hadn't meant to make visible yet!]

CLICK HERE to download Skydoms Character Spreadsheet WITH pictures (6.75MB)

CLICK HERE to download Skydoms Character Spreadsheet WITHOUT pictures (280KB)
Berek         Diamond           ID=948.549.375
Seeking friends with Beelzebub captains.
I have Pride, Anubis, Set, Nezha...


Reply
#10
WOW! This is great, Berek! You have put a lot of time & effort into this. I can appreciate your dedication. Thank you so much for sharing this with the community! Nice work!

(By the way, I see in your signature that you have both Trixie & Brook, the Red & Green Starters. How did you get both of them? I did not think it was possible to possess more than one starter - other than Margot, that is; but I don't count her as a starter.)
Reply




Users browsing this thread: 1 Guest(s)