/
/

Site Navigation

Your Account

Choose Language

Discussion Topic

Options

An offline version of iFixit

Hi there,

I'm currently writing a software to create offline versions of iFixit. These archives are using a popular format named ZIM, browsable with a software named Kiwix. The Kiwix initiative is sponsored by the Wikimedia foundation and has already gained quite a lot of traction. The main interest is for communities or situation where you have no access to Internet (see Kiwix website for details). You can host the ZIM archives you want somewhere, e.g. on a small Raspbery PI Wifi hotspot et voilà.

My main goal is to create a monthly-updated archive of iFixit, mainly categories and guides to start with, but maybe also other content if meaningful.

I'll let you know once the archives are readily available, but I would love to here from you if you have any advice, recommendation, interest in the topic.

Thanks

PS: I'm not working for Wikimedia or Kiwix, it is a spare time / hobby project

Reply to discussion Subscribe to discussion

Is this a worthwhile discussion?

Yes No
Score 9
Add a comment

4 Replies

Filter by:
Most Helpful Newest Oldest
Most Helpful Answer

Options

Hello! This is fantastic and we love that you're doing it. I agree that it makes perfect sense to have an offline version of all of iFixit. Let us know if you have any questions about the API. Hopefully the documentation is pretty self-explanatory!

We used to do this ourselves but haven't in several years and I'm happy to have the community take it over.

We developed this file standard, oManual, to facilitate this kind of archive. I don't know if ZIM makes more sense / has more tooling out there, but you at least can take a look at that spec and see what we were thinking.

FYI when we've supported this in the past, we put both the oManual files and the PDFs on Archive.org.

We don't have the engineering bandwidth to maintain this ourselves, and would love it if you got this project up and off the ground! Have fun.

Was this reply helpful?

Yes No
Score 12

1 Comment:

Hello, Yes... I know I'm Late.

@kyle - Maybe it could used for a new android Ifixit app!

@benoit74 - Keep Up the good work!

by

Add a comment

Options

Hello,

Good to hear that you find it so fantastic ^^

ZIM format / Kiwix has the advantage to work under the umbrella of the Wikimedia foundation with a mission like "bring as much content as possible available offline". The team / budget is rather small as well (~3 persons if I'm not mistaken), but it is obviously their core mission to develop/maintain the tooling, which I consider is a big advantage. They already have quite a lot of content available. Just to name the most popular one: wikipedia in many many languages and subsets, gutemberg, TED videos, etc ...

In terms of tooling, among many things they have developed a ZIMfarm to run all scrapers on a regular basis, many tooling / library to make new scrapers development smooth (I already have an almost ready version with all categories and guides, just fine tuning it before a 0.1 release), a hotspot to make everything run on a Raspberry Pi for quick deployment in remote locations, an ongoing card shop to prepare an SD card for the hotspot with every content you would like (no need to download the hotspot content + individual ZIM files and place everything at the right place).

And of course they regularly upload everything on Archive.org (or at every scraping, not sure about this).

I will have a look at oManual, but it looks rather specific at first sight, where ZIM format is much more general, it is just a kind of "smart ZIP for web content", which can be processed directly without having to extract everything, stores a full-text index that does not have to be extracted.

Regarding the API I already found some quirks/bugs I reported in the new "Bugs Folder", but I also found some missing information (at least for my needs) which I don't know where to report to be honest. The documentation was anyway very helpful / complete, no issue on that, very good job !

I will let you know once I get something ready to be seen ^^

Was this reply helpful?

Yes No
Score 6

5 Comments:

Make sure you incorporate a sync function so updates besides new sections get downloaded when online. Also have a last time sync'ed timer so the person knows if the data may have gone stale (time to sync)

by

Hi, thank you for your interest @danj

Unfortunately, there won't be any synchronization feature, it is not the way of working of ZIMs + it is a way too complex topic (you have to ensure to not overwhelm iFixit servers, you have to handle nicely abrupt deconnexion or software shutdown, you need to be able to update your offline archive on-the-fly, etc ...)

ZIMs are however refreshed on a regular basis, typically once a month, so that you can always download a rather current version when you get back online. Missing at most the last month of edits is usually not a huge deal for users who are used to have no Internet access at all.

From what Kiwix team has experienced with most offline users, this is the preferred way of most of them, since it allows them to retrieve the archive by many simple ways (it is just a file to download, you can ask someone for it, you can forward/share it peer to peer even offline via thumb drives, DVDs, ...).

I understand this might not match your need, but for now this will be.

by

@benoit74 - Can't you access a version ID if we had one? Polling for a hierarchal version so you have a base ID which is altered when the higher levels hav a change so as an example the MacBook's haven't had any changes so looking deeper is not needed, then if it had an update the next level would be checked and on down. This can all be handled in one data segment per grouping. Read up on now Lotus Notes IDs work.

by

I could, definitely, but as I said it is a way too complex topic from my point of view. As of today, I'm a solo dev doing it on my spare time. I prefer to have an archive quickly available after few days of development thanks to Kiwix already available tooling + updated monthly as all ZIMs today, rather than start a probably never ending effort which I won't be able to maintain in the long run as issues will arise. It is already a demanding task to maintain a monthly scraper, iFixit themselves gave up, see Kyle comment above.

by

And by the way, I don't mind if you start another solution which is always updating, it is just not what I feel inclined to do, but like they said, "you say it, you own it" ^^

by

Add a comment

Options

Hello everyone !

The archives are on their way, the scrapper is finally ready to go.

In a few days, I will be able to notify you about the availability of all archives in the 12 languages supported by iFixit.

Categories, guides, comments and wiki pages will be available for offline reading, as well as user pages for everyone who contributed something.

Archives will be automatically updated 4 times per year to retrieve new guides / products.

As I said before, those archives are standalone, so once you've downloaded them and installed them in a Kiwix reader / hotspot, you are free to go.

@kyle, should we prepare something to post on the blog to advertise it a little bit / explain how it works? I'm based in France so already a little bit in contact with @claire17, we might prepare something together if you both agree that it makes sense for you. If possioble, I would also enjoy have a small discussion with a technical person (lead dev, CTO, whatever) to check that we do not load too much your servers / database during the scraping / discuss about possible solutions / investigate alternatives to make the scraper run even faster (it takes about 2 days to build an archive for now).

Cheers,

Benoît

Was this reply helpful?

Yes No
Score 3

1 Comment:

@benoit74 Thanks for the great update! I answered you per email :)

by

Add a comment

Options

By the way, I also had a look at oManual, and the phylosophy is clearly different.

OManuals are clearly more machine friendly since you have a nice structure describing the stuff, but at the expense to have to maintain this specific format + reader + ...

Kiwix is clearly more versatile since you only store assets in the archive (html, css, images, ...) and so the effort to maintain it is spread across many initiatives. But clearly the format is not meant to be processed by a machine.

If many manual providers would come on the market, then oManuals could be a solution to build a shared format across all providers.

I won't have the bandwidth to maintain it myself, clearly ^^ I already have some doubts about my ability to maintain the kiwix scrapper in a good shape for many years :D

Was this reply helpful?

Yes No
Score 2

3 Comments:

I've tried wikipedia offline in the past, can't wait to see the result of iFixit on kiwix.

by

@es_six - Keep Up the good work!

by

@benoit74 I have an idea for you,

If bandwidth is a problem, you should try to host the required files via torrent protocol, some linux distro provide this kind of download method (and also Wikipedia itself) so I will be able to help you reduce the bandwidth usage by setting up a server with a torrent client to host and share files with you at the same time (and also all users that will download these files).

Limitations and possibilities :

- Max available storage space : 150GB

- Max bandwith per month : ~2TB

- Max upload speed : ~300MB/s

- Availability not granted (approximately 90% uptime)

- Automation to downlaod and host latest version : possible if you provide a .torrent file or a magnet link and a fixed URL scheme (or a rest API) to download this information.

With torrent protocol, no need to do a complicated hosting setup, there is plenty of open (and free) trackers on the net.

by

Add a comment

Join the discussion

Benoit74 will be eternally grateful.