From 02dc6423e5de62888dfb6ff549ad22daeb087180 Mon Sep 17 00:00:00 2001 From: Arpit Gogia Date: Wed, 30 Aug 2017 20:27:06 +0530 Subject: [PATCH 1/3] Added writeup for WorldBrain's Webmemex --- writeups/worldbrain_webmemex_arpitgogia.md | 176 +++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 writeups/worldbrain_webmemex_arpitgogia.md diff --git a/writeups/worldbrain_webmemex_arpitgogia.md b/writeups/worldbrain_webmemex_arpitgogia.md new file mode 100644 index 0000000..1d033f9 --- /dev/null +++ b/writeups/worldbrain_webmemex_arpitgogia.md @@ -0,0 +1,176 @@ +# WorldBrain's WebMemex + +A browser extension that lets you create, search and share your personal web of knowledge. + + - **Full-Text Search** pages and you visited and bookmarked (and later) also all the apps you use to organise your knowledge. + - **Manually and automatically create links** between your content to organise & search your web by *your associations* + - **Share** content recommendations, associations and qualifying metadata with your network and followers + + +## Project status + +Sorry to disappoint, but we do not have the promised solution complete yet. :unamused: + +In its current released version, the you can full-text search the browsing history & bookmarks. +Here you can download it: [worldbrain.io/download](http://worldbrain.io/download) + +At the moment we are in the process of refactoring/porting the features of the [WorldBrain (Re)search-Engine](https://github.com/WorldBrain/Research-Engine/) into the *WebMemex*, a project started by [@treora](https://github.com/Treora) + +This project is in full development. You are most welcome to contribute! See +[Hacking](#hacking) below about how to build and run it from source and teach it +new tricks. + + +## Approach + +As it stands now, it is a WebExtension (thus should work on most modern +browsers), bundled by [browserify](http://browserify.org) with some +[babel](https://babeljs.io) ES6–7→ES5 compilation, that logs and stores visited +pages in [PouchDB](https://pouchdb.com), and provides a viewer for this data +based on +[React](https://facebook.github.io/react/)+[Redux](http://redux.js.org/). See +[Code Anatomy](#code-anatomy) below for a full tour du code. + +The project strategy is to combine and integrate features from other projects, +and to factor out developed functionality into separate modules wherever it +seems sensible. This extension could then be regarded as a bunch of different +(but related) features a browser ought to have, bundled together for quick +installation. + +See [our initial blog post](https://blog.webmemex.org/2017/01/05/roadmap/) +for the feature roadmap of the next few months. + + +## Contribute + +We are happy about any kind of feedback, bug fixes, new feature ideas and tips? +Give a shout. :loudspeaker: + +- [Drop by on Slack](http://join-worldbrain.herokuapp.com) +- Write us an email: info@worldbrain.io +- or leave a comment under issues you want to contribute to + + +## Hacking + +Like playing with ES6, WebExtension browser APIs, React, Redux, PouchDB? Come +play along! :tada: + +### Build and run it + +1. Clone this repo. +2. Get [Node/NPM](https://nodejs.org) and [yarn](https://yarnpkg.com) + (`npm install -g yarn`). +3. Run `yarn` to install dependencies. +4. Compile the source files using `yarn watch` for incremental builds or `yarn build` for standard dev builds. +5. Load it in Firefox or Chromium/Chrome: + * In Firefox (≥49): run `yarn run firefox` (or run [web-ext] directly for more control). + Alternatively, go to `about:debugging`, choose 'Load Temporary Add-on', and pick + `extension/manifest.json` from this repo. + * In Chromium/Chrome: go to Tools→Extensions (`chrome://extensions`), enable 'Developer mode', + click 'Load unpacked extension...', and pick the `extension/` folder from this repo. + +### Automatic recompilation + +If the steps above worked, running `yarn run watch` will trigger a quick +recompilation every time a source file has been modified. + +If you are testing in Firefox through `yarn run firefox`, the extension should also reload +automatically. Otherwise, manually press the reload button in the extension list. + +- If your edits affected only the overview interface, just refresh/reopen it. +- However, if you changed the background script, you have will to reload the + extension: find it back in the list of temporary add-ons/extensions and click + Reload. +- If you changed the 'content_script', it seems browser-dependent whether newly + opened pages will get the new version. Better reload the extension to be sure. + + +## Code Anatomy + +### WebExtension parts + +To comply with the [anatomy of a WebExtension](https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Anatomy_of_a_WebExtension), +it consists of three main parts (found in [`extension/`](extension/) after +compilation): + +- `background.js` always runs, in an 'empty invisible tab', listening for + messages and events. +- `content_script.js` is loaded into every web page that is visited. It is + invisible from that web page's own scripts, and can talk to the background + script. +- `overview/overview.html`, with the resources in that folder, provides the main + user interface. + +The parts communicate in two ways: +- Messaging through `browser.sendMessage`, usually done implicitly by using a + remote procedure call ([`util/webextensionRPC.js`](src/util/webextensionRPC.js)). +- Through the in-browser PouchDB database, they get to see the same data, and + can react to changes made by other parts. + +Besides these parts, +[`browser-polyfill.js`](https://github.com/mozilla/webextension-polyfill/) +provides the promise-based `browser` API, that simply wraps Chromium/Chrome's +callback-based `chrome` API, in order to make the same code run in different +browsers (and to structure the callback mess). + +### Source organisation + +To keep things modular, the source code in [`src/`](src/) is not split in +exactly those the three parts of the extension, but is rather grouped by +functionality. Some folders may end up being factored out into separate repos +later on, or at some point perhaps even into separate but interacting browser +extensions. + +#### [`src/activity-logger/`](src/activity-logger/): activity logger + +This logs every page visit in PouchDB. Soon it should also watch for user +interactions, for example to remember which parts of a page you have read. + +Currently, for every visit, a new page object is created in the database, to +represent the visited page itself. This object should soon be deduplicated when +the same page is visited multiple times. After creating a new page object, +the next module is triggered to start analysing the page. + +#### [`src/page-analysis`](src/page-analysis/): page analysis + +This extracts and stores information about the page in a given tab, such as: +- The plain text of the page, mainly for the full-text +search index. +- Metadata, such as its author, publication date, etcetera. +- A screenshot for visual recognition. + +#### [`src/overview/`](src/overview/): overview + +The `overview` is the user interface that opens in a tab of its own. It is built +with [React](https://facebook.github.io/react/) and [Redux](http://redux.js.org/), +which create a somewhat complex but nicely organised application structure. + +See [`src/overview/Readme.md`](src/overview/Readme.md) for more details. + +#### [`src/search`](src/search/): document search + +Functions for finding relevant knowledge in the user's memory. Currently +provides a simple full-text keyword search through visited pages using +[pouchdb-quick-search](https://github.com/nolanlawson/pouchdb-quick-search). +This can be improved in many ways, because we are searching through a person's +memory, not just some arbitrary document collection. For example, we can use +created assocations and browsing paths to better understand what one is looking +for. + +#### [`src/dev/`](src/dev/): development tools + +Tools to help during development. They are not used in production builds. + +#### [`src/util/`](src/util): utilities + +Contains small generic things, stuff that is not project-specific. Things that +could perhaps be packaged and published as an NPM module some day. + +#### `...`: other stuff + +The build process is based on `yarn`, that runs some `npm` commands specified in +`package.json`, which in turn start the corresponding tasks in +`gulpfile.babel.js` (transpiled by settings in `.babelrc`). + +So much for the code tour. :zzz: Any questions? :point_up: From d5608235224e2b77e14fd15918b649b5cff870b2 Mon Sep 17 00:00:00 2001 From: Arpit Gogia Date: Wed, 30 Aug 2017 20:28:00 +0530 Subject: [PATCH 2/3] Added WorldBrain's Webmemex to the list --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 6faca9a..8ba0818 100644 --- a/README.md +++ b/README.md @@ -10,3 +10,4 @@ Please follow the [contributing guide](CONTRIBUTING.md) to add your own contribu | [Ceph](http://ceph.com/) | [Multilanguage RGW Testing](writeups/Ceph_Multilaguage-RGW-Testing_JoannahNanjekye.md) | [Joannah Nanjekye](https://github.com/nanjekyejoannah) | | [Cloud Native Computing Foundation](https://www.cncf.io/) | [Kubernetes](writeups/CNCF_Kubernetes_NikhitaRaghunath.md) | [Nikhita Raghunath](https://github.com/nikhita) | | [phpMyAdmin](https://www.phpmyadmin.net/) | [phpMyAdmin](writeups/phpmyadmin_phpmyadmin_manishbisht.md) | [Manish Bisht](https://github.com/manishbisht) | +| [WorldBrain](https://worldbrain.io/) | [WebMemex](writeups/worldbrain_webmemex_arpitgogia.md) | [Arpit Gogia](https://github.com/arpitgogia) | From 21e6201286925a2f1fced3cca20f9d4a01e5ebb9 Mon Sep 17 00:00:00 2001 From: Arpit Gogia Date: Wed, 30 Aug 2017 21:08:31 +0530 Subject: [PATCH 3/3] Grammatical and spelling fixes --- writeups/worldbrain_webmemex_arpitgogia.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/writeups/worldbrain_webmemex_arpitgogia.md b/writeups/worldbrain_webmemex_arpitgogia.md index 1d033f9..15e02d6 100644 --- a/writeups/worldbrain_webmemex_arpitgogia.md +++ b/writeups/worldbrain_webmemex_arpitgogia.md @@ -2,7 +2,7 @@ A browser extension that lets you create, search and share your personal web of knowledge. - - **Full-Text Search** pages and you visited and bookmarked (and later) also all the apps you use to organise your knowledge. + - **Full-Text Search** pages that you visited and bookmarked (and later) also all the apps you use to organise your knowledge. - **Manually and automatically create links** between your content to organise & search your web by *your associations* - **Share** content recommendations, associations and qualifying metadata with your network and followers @@ -11,14 +11,13 @@ A browser extension that lets you create, search and share your personal web of Sorry to disappoint, but we do not have the promised solution complete yet. :unamused: -In its current released version, the you can full-text search the browsing history & bookmarks. -Here you can download it: [worldbrain.io/download](http://worldbrain.io/download) +In its current released version, you can full-text search the browsing history & bookmarks. +You can download it here: [worldbrain.io/download](http://worldbrain.io/download) At the moment we are in the process of refactoring/porting the features of the [WorldBrain (Re)search-Engine](https://github.com/WorldBrain/Research-Engine/) into the *WebMemex*, a project started by [@treora](https://github.com/Treora) This project is in full development. You are most welcome to contribute! See -[Hacking](#hacking) below about how to build and run it from source and teach it -new tricks. +[Hacking](#hacking) below about how to build and run it from source and make it learn new tricks. ## Approach @@ -43,7 +42,7 @@ for the feature roadmap of the next few months. ## Contribute -We are happy about any kind of feedback, bug fixes, new feature ideas and tips? +We are happy about any kind of feedback, bug fixes, new feature ideas and tips! Give a shout. :loudspeaker: - [Drop by on Slack](http://join-worldbrain.herokuapp.com) @@ -155,7 +154,7 @@ provides a simple full-text keyword search through visited pages using [pouchdb-quick-search](https://github.com/nolanlawson/pouchdb-quick-search). This can be improved in many ways, because we are searching through a person's memory, not just some arbitrary document collection. For example, we can use -created assocations and browsing paths to better understand what one is looking +created associations and browsing paths to better understand what one is looking for. #### [`src/dev/`](src/dev/): development tools