Shred Shoe Finder

[Allan]

Good people like MetZelRio have been posting on the Facebook group when they find potentially good shred shoes on eBay. I got the idea a couple of nights ago that it would be great if we could keep those posts towards the surface of things, as not everyone checks Facebook obsessively. Of course, Facebook doesn't really let this happen, and posts get pushed out of sight pretty quickly. Using the forums for this is problematic too, as it takes a fair amount of effort to maintain up-to-date lists in a top-post, unnecessarily burdening a single person. As such, I created:

The Shred Shoe Finder

I'm posting it in here first as I want to solicit feedback from you guys on how it should work. At the moment, this is eBay-only. I will be expanding it so that you can create stand-alone listings, but that's a ways off yet. It's pretty obvious how this works: after you've logged in to modified, you can visit the shred shoe finder and simply paste your eBay link into the field and hit go. My script grabs the main photo, title, and price and stores it in a database and spits it out with the rest of them on the page, sorted in reverse chronological order.

This is currently the result of two nights half-working (watching The Knick right now; so gory, but so good) on this, so it's pretty rough. There are a lot of different URL schemas that eBay uses for listings, so parsing things correctly is a game of whack-a-mole; some URLs you might paste might not work.

As I'm not, actually, using the eBay API for this (I'm manually scraping the page), getting the price of the shoes is also somewhat problematic. If this thing actually gets any use, I'd consider making it more robust by actually using the proper way; in the meantime, I'm super lazy

Features to come:

* the moderator group will get an email when a new link is posted, and/or when a regular user flags a listing (see below)
* moderators are able to actually edit/remove listings
* regular logged-in users will be able to flag a shoe as "sold" or whatever
* some form of categorization, likely by size, then maybe by type (laver, goonit, etc)

This would get advertised on the forum via a banner when it comes time to implement that (when I get the logo from Holly), and could be a sticky post in, like, the equipment forum or whatever when it's more finished.

Interested in helping? Join the moderator group!

Thoughts?

PS let's not post this on the Facebook group just yet, if that's OK. I'd like some of the above features in place before more people see it...

[F[uns]tylin' Eclectic]

Awesome idea, Allan. I will check it out further when I get home

[Kylescook]

I think this is a great idea Allan. I would be happy to help in any way I can.

[Allan]

Thanks for chiming in, peeps I wrote a bit more code on this, and introduced a couple of fields for size and category; mods can now remove shoes. I've been letting this ruminate over the last day or so, and I'll put in a bit more on it this week.

[Allan]

I made some decent progress on this last night

On a few other projects, I've been using this awesome script for in-page searching and filtering; it's wicked fast, filter as-you-type search. So you can now sort on price and size, as well as doing searches like "green" or "unit"

I made the link parser a bit more robust, but that's going to be an ongoing thing. I am absolutely astounded that there is not already an eBay URL parsing library to make this easier; it's a bit weird really. I also wanted to automatically convert shoe sizes for the lovely people outside of North America, but, amazingly, there doesn't appear to be any sort of open source function to do this; I must simply be looking in the wrong places. UPDATE: Found something! Cool.

I've got a couple of ideas on how to make this better, but I think perhaps my energies would be better spent elsewhere. If people actually end up using this, I'll put more effort into it. As it is, I think I'll probably put another session or two into it, and then post about it elsewhere.

Anybody have any feedback?

[Kylescook]

Allan,

Thanks so much for this. It actually tipped me off to a Pair of old Lavers I won in an auction last night. I will be checking it every day or so for outdated listings. I noticed there was a listing for a newer model of Lavers, Should I remove those as well when I come across them?

[Allan]

Kyle, yeah, probably best to remove any new-edition lavers you come across. Honestly, I didn't give the listings that are there right now a thorough vet before posting them; they were just data to work with that I figured would work well enough to begin with Good lookin' out. Thanks.

[Kylescook]

Hey Allan,

I noticed the option to remove shoes from the shred shoe finder is no longer there. If I'm missing something please let me know.

[Allan]

Bottom right? It's still there for me ... You do have to be logged in to modified to see it.

Thanks for your help with this.

I've not had much time over the past couple of weeks for this or other fizzie stuff ... hopefully I'll be able to finish this off soon

[MetZelRio]

Embarrassed to say, I didnt realize this has been operational, even though I signed up to be a mod. I made up for it a little. I added one zillion shoes this morning and still working on it. Wow, is this great or what? Its like our own shoe store with (hopefully) no non shredable shoes.

[Kylescook]

I guess I may not have been logged into modified. Thanks Allan.
Edit: Also, Great job on all the added shoes Paul.

[jaust]

wow this is great!

haven't been on modified for a while, but immediately found this and added a few pairs of rod lavers and g's

thanks for the work allan!

[Kylescook]

Thanks for helping out.

[Allan]

I've been wanting to give this a bit more love, but I'm being pulled in a bunch of different directions right now. Thanks for the help adding & removing shoes jaust Kylescook and MetZelRio I appreciate it tons.

What would be super cool would be if I were actually implement the eBay API, then I could automagically programmatically check for outdated listings on a periodic basis and remove them without any human intervention... hopefully inspiration will strike on that soonish; in the meantime, thanks for help everyone.

I'm thinking that maybe this should be posted to the Facebook group soon, even though it's not exactly perfect. Thoughts?

[Kylescook]

I'm thinking that maybe this should be posted to the Facebook group soon, even though it's not exactly perfect. Thoughts?

I think It functions well enough use. Plus as more people become aware of it more people will likely add shoes when they see them. The self expiring posts would be a neat addition.

[jaust]

what if you would simply give the posts in the shoe finder an average expiration date, like 14 days or something. since an ebay auction usually does not run longer anyways... if it's to much work to use the exact date, instead just set the expiration date yourself.

[Allan]

what if you would simply give the posts in the shoe finder an average expiration date, like 14 days or something. since an ebay auction usually does not run longer anyways... if it's to much work to use the exact date, instead just set the expiration date yourself.

I think this might create more work than it's worth, since the expiration times are actually extremely variable; for example, those climacools have been valid listings for months now. If I set an arbitrary date, there might be listings that get removed prematurely. As it is, the script is already sorting on a reverse chronological order (newest first), so mods really only have to start at the bottom (oldest) and work their way up. This is definitely not ideal, but if I'm going to code a solution, I'd rather have it be the proper one (using the API) than a hack (on top of a hack) that doesn't really work all that reliably. Ya dig?

Using the API isn't exactly difficult. I should really just do it. Honestly, however, I wasn't overly stoked on the rather weak response from the Facebook group. 30 likes out of 1500+ people means that either not a lot of people saw it, or not a lot of people care enough for me to put much more work into it. I could make it an academic exercise ...

[isirc10]

I think people might not realize how awesome this is until they actually need a new pair of lavers/units. Personally, I only checked it out now and think it's really useful. And lets be honest, there are like 100 people who post/link/comment on that group. The rest either lurk or have forgotten that they joined. I say keep working on it, if it's interesting/fun for you.

[Allan]

I think people might not realize how awesome this is until they actually need a new pair of lavers/units.

Yeah.

Personally, I only checked it out now and think it's really useful.

Cool!

And lets be honest, there are like 100 people who post/link/comment on that group. The rest either lurk or have forgotten that they joined. I say keep working on it, if it's interesting/fun for you.

You're totally right. I must have just been in a bad mood the other day when I wrote that I wasn't stoked on the response. I seriously don't do _any_ of this stuff for the likes or the comments or whatever. It would have been nice to get a better response, but I know that it's already helped a few people out, and that's all that actually matters to me. If it can continue to help some folks out, that should be motivation enough.

I was sucking at code tonight and spent hours and hours on one thing that I just couldn't make work (forcing Wordpress to use SFTP for updates on my server). I ended up giving up on that, but I didn't want to go down completely defeated for the day, so I finally took a close look at the eBay API documentation. After an hour or so of hacking on it, I totally got it working. It's not implemented yet as I simply got the API to respond properly to test queries and was able to parse the response and I'm now completely out of steam for the day. But it's all downhill from here and within the next day or two, the list should be self-moderating for expired sales! I'll set it to check daily and any shoes that get sold or whatever will be automagically removed from the page So at least I did something right today

More soon...

[Allan]

Aaaaaand it's running I was still stoked on it when I got up, so I put a couple more hours into the code and got it fully working. It removed a couple of listings this morning already, and will update every 6 hours to make sure that everything is up to date and active. I still need to code some robustness into the script, but I'm stoked that we're finally using the API and our wonderful moderators no longer need to manually look at every listing to ensure that things are valid. Now they can focus their energies on making sure that there are no shitty shoes listed

My next steps are to use my newfound API powers to make the import process more robust (e.g. removing the need to input the shoe size for starters) and then I'll probably clean things up a bit and that should pretty much be that. I'm sure I'll come up with some other enhancements soon. If anyone has any ideas of how to make it cooler, do speak up