Version 4.0.8

Closes #54

.github/ISSUE_TEMPLATE/bug_report.md

@@ -7,6 +7,11 @@ assignees: ''
 
 ---
 
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
 **Describe the bug**
 A clear and concise description of what the bug is.
 
@@ -23,16 +28,6 @@ A clear and concise description of what you expected to happen.
 **Screenshots**
 If applicable, add screenshots to help explain your problem.
 
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
-
-**Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
 
 **Additional context**
 Add any other context about the problem here.

Readme.md

@@ -1,12 +1,10 @@
-DownThemAll! WE
-===
+# DownThemAll! WE
 
 The DownThemAll! WebExtension.
 
 For those still on supported browser: [Non-WebExtension legacy code](https://github.com/downthemall/downthemall-legacy).
 
-About
----
+## About
 
 This is the WebExtension version of DownThemAll!, a complete re-development from scratch.
 Being a WebExtension it lacks a ton of features the original DownThemAll! had. Sorry, but there is no way around it since Mozilla decided to adopt WebExtensions as the *only* extension type and WebExtensions are extremely limited in what they can do.
@@ -23,25 +21,48 @@ But it is what it is...
 
 **What we *can* do and did do is bring the mass selection, organizing (renaming masks, etc) and queueing tools of DownThemAll! over to the WebExtension, so you can easily queue up hundreds or thousands files at once without the downloads going up in flames because the browser tried to download them all at once.**
 
-Translations
----
+## Translations
 
 If you would like to help out translating DTA, please see our [translation guide](_locales/Readme.md).
 
-Development
----
+## Development
+
 
 You will want to `yarn` the development dependencies such as webpack first.
 
-Afterwards there is two important commands to run
+Afterwards, you will want to run`yarn watch`.
+This will run the webpack bundler in watch mode, transpiling the TypeScript to Javascript and updating bundles as you change the source.
 
-  * `yarn watch` - This will run the webpack bundler in watch mode, updating bundles as you change the source.
-  * `yarn webext` - This will run the WebExtension in a development profile using the [`web-ext` tool from mozilla](https://www.npmjs.com/package/web-ext) (which you need to install separately). This will use the directory `../dtalite.p` to keep a development profile. You might need to create this directory before you use this command first.
-  
 Please note: You have to run `yarn watch` (at least once) as it builds the actual script bundles.
+
+### Firefox
+
+I recommend you install the [`web-ext`](https://www.npmjs.com/package/web-ext) tools from mozilla. It is not listed as a dependency by design at it causes problems with dependency resolution in yarn right now if installed in the same location as the rest of the dependencies.
+
+If you did, then running `yarn webext` (additionally to `yarn watch`) will run the WebExtension in a development profile. This will use the directory `../dtalite.p` to keep a development profile. You might need to create this directory before you use this command. Furthermore `yarn webext` will watch for changes to the sources and automatically reload the extension.
   
-Alternative, you can also `yarn build`, which then builds an *unsigned* zip that you can then install permanently in a browser that does not enforce signing (i.e. Nightly or the Unbranded Firefox).
+Alternative, you can also `yarn build`, which then builds an *unsigned* zip that you can then install permanently in a browser that does not enforce signing (i.e. Nightly or the Unbranded Firefox with the right about:config preferences).
 
-Before submitting patches, please make sure you run eslint, if this isn't done automatically, and eslint does not report any open issues. Code contributions should favor typescript code over javascript code. External dependencies that would ship with the final product (including all npm/yarn packages) should be kept to a bare minimum.
+### Chrome
 
-The code base is comparatively large for a WebExtension, with over 10K sloc of typescript and over 14K sloc total.
+You have to build the bundles first, of course.
+
+Then put your Chrome into Developement Mode on the Extensions page, and Load Unpacked the directory of your downthemall clone.
+
+### Patches
+
+Before submitting patches, please make sure you run eslint (if this isn't done automatically in your text editor/IDE), and eslint does not report any open issues. Code contributions should favor typescript code over javascript code. External dependencies that would ship with the final product (including all npm/yarn packages) should be kept to a bare minimum and need justification.
+
+Please submit your patches as Pull Requests, and rebase your commits onto the current `master` before submitting.
+
+### Code structure
+
+The code base is comparatively large for a WebExtension, with over 11K sloc of typescript.
+It isn't as well organized as it should be in some places; hope you don't mind.
+
+* `uikit/` - The base User Interface Kit, which currently consists of
+  * the `VirtualTable` implementation, aka that interactive HTML table with columns, columns resizing and hiding, etc you see in the Manager, Select and Preferences windows/tabs
+  * the `ContextMenu` and related classes that drive the HTML-based context menus
+* `lib/` - The "backend stuff" and assorted library routines and classes.
+* `windows/` - The "frontend stuff" so all the HTML and corresponding code to make that HTML into something interactive
+* `style/` - CSS and images

TODO.md

@@ -15,10 +15,6 @@ Planned for later.
   * Add downloads
 * Chrome support
 * vtable perf: cache column widths
-* Localizations
-  * Settle on system
-  * Do the de-locale
-  * Enagage translators
 * Download options
   * This is a bit more limited, as we cannot modify options of downloads that have been started (and paused) or that are done.
 
@@ -39,7 +35,6 @@ Nice-to-haves.
 * Dark Theme support
   * os/browser define be default
   * overwritable
-* Get and cache system icons (because Firefox doesn't allow moz-icon: for WE, but makes them kinda accessible through the downloads API anyway, essentially copying them via a canvas on a privileged hidden page into a data URL... ikr)
 * Remove `any` types as possible, and generally improve typescript (new language to me)
 
 P4

_locales/all.json

@@ -0,0 +1,17 @@
+{
+  "cs": "Čeština (CZ) [cs]",
+  "de": "Deutsch [de]",
+  "el": "Ελληνικά [el]",
+  "en": "English (US) [en]",
+  "es": "Español (España) [es]",
+  "et": "Eesti Keel [et]",
+  "fr": "Français (FR) [fr]",
+  "id": "Bahasa Indonesia [id]",
+  "ko": "한국어 [ko]",
+  "lt": "Lietuvių [lt]",
+  "nl": "Nederlands [nl]",
+  "pl": "Polski (PL) [pl]",
+  "pt": "Português (Brasil) [pt]",
+  "ru": "Русский [ru]",
+  "zh_CN": "简体中文 [zh_CN]"
+}

lib/background.ts

@@ -17,7 +17,8 @@ import {
   // eslint-disable-next-line no-unused-vars
   Tab,
   // eslint-disable-next-line no-unused-vars
-  MenuClickInfo
+  MenuClickInfo,
+  CHROME,
 } from "./browser";
 import { Bus } from "./bus";
 import { filterInSitu } from "./util";
@@ -27,6 +28,20 @@ const menus = typeof (_menus) !== "undefined" && _menus || _cmenus;
 
 const GATHER = "/bundles/content-gather.js";
 
+const CHROME_CONTEXTS = Object.freeze(new Set([
+  "all",
+  "audio",
+  "browser_action",
+  "editable",
+  "frame",
+  "image",
+  "launcher",
+  "link",
+  "page",
+  "page_action",
+  "selection",
+  "video",
+]));
 
 async function runContentJob(tab: Tab, file: string, msg: any) {
   try {
@@ -83,11 +98,15 @@ class Handler {
 
   async performSelection(options: SelectionOptions) {
     try {
+      const toptions: any = {
+        currentWindow: true,
+        discarded: false,
+      };
+      if (!CHROME) {
+        toptions.hidden = true;
+      }
       const selectedTabs = options.allTabs ?
-        await tabs.query({
-          currentWindow: true,
-          discarded: false,
-          hidden: false}) as any[] :
+        await tabs.query(toptions) as any[] :
         [options.tab];
 
       const textLinks = await Prefs.get("text-links", true);
@@ -146,10 +165,18 @@ locale.then(() => {
       this.onClicked = this.onClicked.bind(this);
       const alls = new Map<string, string[]>();
       const mcreate = (options: any) => {
+        if (CHROME) {
+          delete options.icons;
+          options.contexts = options.contexts.
+            filter((e: string) => CHROME_CONTEXTS.has(e));
+          if (!options.contexts.length) {
+            return;
+          }
+        }
         if (options.contexts.includes("all")) {
           alls.set(options.id, options.contexts);
         }
-        return menus.create(options);
+        menus.create(options);
       };
       mcreate({
         id: "DTARegularLink",
@@ -389,7 +416,10 @@ locale.then(() => {
     }
 
     async enumulate(action: string) {
-      const tab = await tabs.query({active: true});
+      const tab = await tabs.query({
+        active: true,
+        currentWindow: true,
+      });
       if (!tab || !tab.length) {
         return;
       }
@@ -517,13 +547,21 @@ locale.then(() => {
 
   function adjustAction(globalTurbo: boolean) {
     action.setPopup({
-      popup: globalTurbo ? "" : null
+      popup: globalTurbo ? "" : "/windows/popup.html"
     });
     action.setIcon({
       path: globalTurbo ? {
         16: "/style/button-turbo.png",
         32: "/style/button-turbo@2x.png",
-      } : null
+      } : {
+        16: "/style/icon16.png",
+        32: "/style/icon32.png",
+        48: "/style/icon48.png",
+        64: "/style/icon64.png",
+        96: "/style/icon96.png",
+        128: "/style/icon128.png",
+        256: "/style/icon256.png"
+      }
     });
   }
 

lib/browser.ts

@@ -50,3 +50,5 @@ export const {storage} = polyfill;
 export const {tabs} = polyfill;
 export const {webNavigation} = polyfill;
 export const {windows} = polyfill;
+
+export const CHROME = navigator.appVersion.includes("Chrome/");

lib/filters.ts

@@ -94,7 +94,7 @@ function *parseIntoRegexpInternal(str: string): Iterable<RegExp> {
 
   // multi-expression
   if (str.includes(",")) {
-    for (const part in str.split(",")) {
+    for (const part of str.split(",")) {
       yield *parseIntoRegexpInternal(part);
     }
     return;

lib/i18n.ts

@@ -2,6 +2,19 @@
 // License: MIT
 
 import {memoize} from "./memoize";
+import * as langs from "../_locales/all.json";
+import { sorted, naturalCaseCompare } from "./sorting";
+
+
+export const ALL_LANGS = Object.freeze(new Map<string, string>(
+  sorted(Object.entries(langs), e => {
+    return [e[1], e[0]];
+  }, naturalCaseCompare)));
+
+let CURRENT = "en";
+export function getCurrentLanguage() {
+  return CURRENT;
+}
 
 declare let browser: any;
 declare let chrome: any;
@@ -9,6 +22,8 @@ declare let chrome: any;
 const CACHE_KEY = "_cached_locales";
 const CUSTOM_KEY = "_custom_locale";
 
+const normalizer = /[^A-Za-z0-9_]/g;
+
 interface JSONEntry {
   message: string;
   placeholders: any;
@@ -72,7 +87,7 @@ class Localization {
   }
 
   localize(id: string, ...args: any[]) {
-    const entry = this.strings.get(id);
+    const entry = this.strings.get(id.replace(normalizer, "_"));
     if (!entry) {
       return "";
     }
@@ -119,16 +134,36 @@ function loadCached() {
 }
 
 async function loadRawLocales() {
-  // en is the base locale
+  // en is the base locale, always to be loaded
+  // The loader will override string from it with more specific string
+  // from other locales
   const langs = new Set<string>(["en"]);
-  const ui = (browser.i18n || chrome.i18n).getUILanguage();
-  langs.add(ui);
-  if (ui.includes("-")) {
-    // Try the base too
-    langs.add(ui.split(/[_-]+/)[0]);
+
+  const uiLang: string = (typeof browser !== "undefined" ? browser : chrome).
+    i18n.getUILanguage();
+
+  // Chrome will only look for underscore versions of locale codes,
+  // while Firefox will look for both.
+  // So we better normalize the code to the underscore version.
+  // However, the API seems to always return the dash-version.
+
+  // Add all base locales into ascending order of priority,
+  // starting with the most unspecific base locale, ending
+  // with the most specific locale.
+  // e.g. this will transform ["zh", "CN"] -> ["zh", "zh_CN"]
+  uiLang.split(/[_-]/g).reduce<string[]>((prev, curr) => {
+    prev.push(curr);
+    langs.add(prev.join("_"));
+    return prev;
+  }, []);
+
+  if (CURRENT && CURRENT !== "default") {
+    langs.delete(CURRENT);
+    langs.add(CURRENT);
   }
 
-  const fetched = await Promise.all(Array.from(langs, fetchLanguage));
+  const valid = Array.from(langs).filter(e => ALL_LANGS.has(e));
+  const fetched = await Promise.all(Array.from(valid, fetchLanguage));
   return fetched.filter(e => !!e);
 }
 
@@ -136,6 +171,21 @@ async function load(): Promise<Localization> {
   try {
     checkBrowser();
     try {
+      let currentLang: any = "";
+      if (typeof browser !== "undefined") {
+        currentLang = await browser.storage.sync.get("language");
+      }
+      else {
+        currentLang = await new Promise(
+          resolve => chrome.storage.sync.get("language", resolve));
+      }
+      if ("language" in currentLang) {
+        currentLang = currentLang.language;
+      }
+      if (!currentLang || !currentLang.length) {
+        currentLang = "default";
+      }
+      CURRENT = currentLang;
       // en is the base locale
       let valid = loadCached();
       if (!valid) {

lib/iconcache.ts

@@ -0,0 +1,162 @@
+"use strict";
+// License: MIT
+
+import { downloads, CHROME } from "./browser";
+import { EventEmitter } from "../uikit/lib/events";
+import { PromiseSerializer } from "./pserializer";
+
+const VERSION = 1;
+const STORE = "iconcache";
+// eslint-disable-next-line no-magic-numbers
+const CACHE_SIZES = CHROME ? [16, 32] : [16, 32, 64, 127];
+
+const BLACKLISTED = Object.freeze(new Set([
+  "",
+  "ext",
+  "ico",
+  "pif",
+  "scr",
+  "ani",
+  "cur",
+  "ttf",
+  "otf",
+  "woff",
+  "woff2",
+  "cpl",
+  "desktop",
+  "app",
+]));
+
+async function getIcon(size: number, manId: number) {
+  const raw = await downloads.getFileIcon(manId, {size});
+  const icon = new URL(raw);
+  if (icon.protocol === "data:") {
+    const res = await fetch(icon.toString());
+    const blob = await res.blob();
+    return {size, icon: blob};
+  }
+  return {size, icon};
+}
+
+const SYNONYMS = Object.freeze(new Map<string, string>([
+  ["jpe", "jpg"],
+  ["jpeg", "jpg"],
+  ["jfif", "jpg"],
+  ["mpe", "mpg"],
+  ["mpeg", "mpg"],
+  ["m4v", "mp4"],
+]));
+
+export const IconCache = new class IconCache extends EventEmitter {
+  private db: Promise<IDBDatabase>;
+
+  private cache: Map<string, string>;
+
+  constructor() {
+    super();
+    this.db = this.init();
+    this.cache = new Map();
+    this.get = PromiseSerializer.wrapNew(8, this, this.get);
+    this.set = PromiseSerializer.wrapNew(1, this, this.set);
+  }
+
+  private async init() {
+    return await new Promise<IDBDatabase>((resolve, reject) => {
+      const req = indexedDB.open(STORE, VERSION);
+      req.onupgradeneeded = evt => {
+        const db = req.result;
+        switch (evt.oldVersion) {
+        case 0: {
+          db.createObjectStore(STORE);
+          break;
+        }
+        }
+      };
+      req.onerror = ex => reject(ex);
+      req.onsuccess = () => {
+        resolve(req.result);
+      };
+    });
+  }
+
+  private normalize(ext: string) {
+    ext = ext.toLocaleLowerCase();
+    return SYNONYMS.get(ext) || ext;
+  }
+
+  // eslint-disable-next-line no-magic-numbers
+  async get(ext: string, size = 16) {
+    ext = this.normalize(ext);
+    if (BLACKLISTED.has(ext)) {
+      return undefined;
+    }
+    const sext = `${ext}-${size}`;
+    let rv = this.cache.get(sext);
+    if (rv) {
+      return rv;
+    }
+    const db = await this.db;
+    rv = this.cache.get(sext);
+    if (rv) {
+      return rv;
+    }
+    return await new Promise<string | undefined>(resolve => {
+      const trans = db.transaction(STORE, "readonly");
+      trans.onerror = () => resolve(undefined);
+      const store = trans.objectStore(STORE);
+      const req = store.get(sext);
+      req.onerror = () => resolve(undefined);
+      req.onsuccess = () => {
+        const rv = this.cache.get(sext);
+        if (rv) {
+          resolve(rv);
+          return;
+        }
+        let {result} = req;
+        if (!result) {
+          resolve(undefined);
+          return;
+        }
+        if (typeof req.result !== "string") {
+          result = URL.createObjectURL(result).toString();
+        }
+        this.cache.set(sext, result);
+        this.cache.set(ext, "");
+        resolve(result);
+      };
+    });
+  }
+
+  async set(ext: string, manId: number) {
+    ext = this.normalize(ext);
+    if (BLACKLISTED.has(ext)) {
+      return;
+    }
+    if (this.cache.has(ext)) {
+      // already processed in this session
+      return;
+    }
+    // eslint-disable-next-line no-magic-numbers
+    const urls = await Promise.all(CACHE_SIZES.map(
+      size => getIcon(size, manId)));
+    if (this.cache.has(ext)) {
+      // already processed in this session
+      return;
+    }
+    for (const {size, icon} of urls) {
+      this.cache.set(`${ext}-${size}`, URL.createObjectURL(icon));
+    }
+    this.cache.set(ext, "");
+    const db = await this.db;
+    await new Promise((resolve, reject) => {
+      const trans = db.transaction(STORE, "readwrite");
+      trans.onerror = reject;
+      trans.oncomplete = resolve;
+      const store = trans.objectStore(STORE);
+      for (const {size, icon} of urls) {
+        store.put(icon, `${ext}-${size}`);
+      }
+    });
+    this.emit("cached", ext);
+  }
+}();

lib/manager/basedownload.ts

@@ -153,6 +153,7 @@ export class BaseDownload {
     rv.destPath = dest.path;
     rv.destFull = dest.full;
     rv.error = this.error;
+    rv.ext = this.renamer.p_ext;
     return rv;
   }
 }

lib/manager/download.ts

@@ -11,13 +11,16 @@ import { BaseDownload } from "./basedownload";
 import { PromiseSerializer } from "../pserializer";
 // eslint-disable-next-line no-unused-vars
 import { Manager } from "./man";
-import { downloads } from "../browser";
+import { downloads, CHROME } from "../browser";
+import { debounce } from "../../uikit/lib/util";
 
 
 const setShelfEnabled = downloads.setShelfEnabled || function() {
   // ignored
 };
 
+const reenableShelf = debounce(() => setShelfEnabled(true), 1000, true);
+
 type Header = {name: string; value: string};
 interface Options {
   conflictAction: string;
@@ -26,7 +29,7 @@ interface Options {
   url: string;
   method?: string;
   body?: string;
-  incognito: boolean;
+  incognito?: boolean;
   headers: Header[];
 }
 
@@ -103,13 +106,15 @@ export class Download extends BaseDownload {
         saveAs: false,
         url: this.url,
         headers: [],
-        incognito: this.private
       };
+      if (!CHROME && this.private) {
+        options.incognito = true;
+      }
       if (this.postData) {
         options.body = this.postData;
         options.method = "POST";
       }
-      if (this.referrer) {
+      if (!CHROME && this.referrer) {
         options.headers.push({
           name: "Referer",
           value: this.referrer
@@ -136,7 +141,7 @@ export class Download extends BaseDownload {
         }
       }
       finally {
-        setShelfEnabled(true);
+        reenableShelf();
       }
       this.markDirty();
     }

lib/manager/man.ts

@@ -12,17 +12,19 @@ import { Prefs } from "../prefs";
 import { _ } from "../i18n";
 import { CoalescedUpdate, mapFilterInSitu, filterInSitu } from "../util";
 import { PromiseSerializer } from "../pserializer";
-import {Download} from "./download";
-import {ManagerPort} from "./port";
-import {Scheduler} from "./scheduler";
-import {Limits} from "./limits";
-import { downloads } from "../browser";
+import { Download } from "./download";
+import { ManagerPort } from "./port";
+import { Scheduler } from "./scheduler";
+import { Limits } from "./limits";
+import { downloads, runtime } from "../browser";
 
 
 const AUTOSAVE_TIMEOUT = 2000;
 const DIRTY_TIMEOUT = 100;
 // eslint-disable-next-line no-magic-numbers
 const MISSING_TIMEOUT = 12 * 1000;
+const RELOAD_TIMEOUT = 10 * 1000;
+
 
 export class Manager extends EventEmitter {
   private items: Download[];
@@ -45,9 +47,12 @@ export class Manager extends EventEmitter {
 
   private scheduler: Scheduler | null;
 
+  private shouldReload: boolean;
+
   constructor() {
     super();
     this.active = true;
+    this.shouldReload = false;
     this.notifiedFinished = true;
     this.items = [];
     this.saveQueue = new CoalescedUpdate(
@@ -91,6 +96,13 @@ export class Manager extends EventEmitter {
     await this.resetScheduler();
     this.emit("inited");
     setTimeout(() => this.checkMissing(), MISSING_TIMEOUT);
+    runtime.onUpdateAvailable.addListener(() => {
+      if (this.running.size) {
+        this.shouldReload = true;
+        return;
+      }
+      runtime.reload();
+    });
     return this;
   }
 
@@ -169,6 +181,14 @@ export class Manager extends EventEmitter {
     }
     this.notifiedFinished = true;
     new Notification(null, _("queue-finished"));
+    if (this.shouldReload) {
+      setTimeout(() => {
+        if (this.running.size) {
+          return;
+        }
+        runtime.reload();
+      }, RELOAD_TIMEOUT);
+    }
   }
 
   addManId(id: number, download: Download) {

lib/manager/port.ts

@@ -61,6 +61,7 @@ export class ManagerPort {
       delete this.manager;
       delete this.port;
     });
+
     this.port.post("active", this.manager.active);
     this.sendAll();
   }

lib/notifications.ts

@@ -12,13 +12,20 @@ const DEFAULTS = {
   message: "message",
 };
 
+const TIMEOUT = 4000;
+
+let gid = 1;
+
 export class Notification extends EventEmitter {
   private notification: any;
 
+  private readonly generated: boolean;
+
   constructor(id: string | null, options = {}) {
     super();
 
-    id = id || "DownThemAll-notification";
+    this.generated = !id;
+    id = id || `DownThemAll-notification${++gid}`;
     if (typeof options === "string") {
       options = {message: options};
     }
@@ -39,11 +46,16 @@ export class Notification extends EventEmitter {
   opened(notification: any) {
     this.notification = notification;
     this.emit("opened", this);
+    if (this.generated) {
+      setTimeout(() => {
+        notifications.clear(notification);
+      }, TIMEOUT);
+    }
   }
 
   clicked(notification: any, button?: number) {
     // We can only be clicked, when we were opened, at which point the
-    // notification id is availablfalse
+    // notification id is available
     if (notification !== this.notification) {
       return;
     }
@@ -52,6 +64,7 @@ export class Notification extends EventEmitter {
       return;
     }
     this.emit("clicked", this);
+    console.log("clicked", notification);
   }
 
   async closed(notification: any) {

lib/windowutils.ts

@@ -8,12 +8,26 @@ import * as DEFAULT_ICONS from "../data/icons.json";
 const DONATE_URL = "https://www.downthemall.org/howto/donate/";
 const MANAGER_URL = "/windows/manager.html";
 
-const IS_CHROME = navigator && navigator.userAgent.includes("Chrome");
-
-
 export async function mostRecentBrowser(): Promise<any> {
-  let window = Array.from(await windows.getAll({windowTypes: ["normal"]})).
-    filter((w: any) => w.type === "normal").pop();
+  let window;
+  try {
+    window = await windows.getCurrent({windowTypes: ["normal"]});
+    if (window.type !== "normal") {
+      throw new Error("not a normal window");
+    }
+  }
+  catch {
+    try {
+      window = await windows.getlastFocused({windowTypes: ["normal"]});
+      if (window.type !== "normal") {
+        throw new Error("not a normal window");
+      }
+    }
+    catch {
+      window = Array.from(await windows.getAll({windowTypes: ["normal"]})).
+        filter((w: any) => w.type === "normal").pop();
+    }
+  }
   if (!window) {
     window = await windows.create({
       url: DONATE_URL,
@@ -106,32 +120,10 @@ const ICONS = Object.freeze((() => {
   return new Map<string, string>(rv);
 })());
 
-let iconForPathPlatform: Function;
-if (IS_CHROME) {
-  const FOUR = 128;
-  const DOUBLE = 64;
-  iconForPathPlatform = function(icon: string, size: number) {
-    let scale = "1x";
-    if (size > FOUR) {
-      // wishful thinking at this point
-      scale = "4x";
-    }
-    else if (size > DOUBLE) {
-      scale = "2x";
-    }
-    return `chrome://fileicon/${icon}?scale=${scale}`;
-  };
-}
-else {
-  // eslint-disable-next-line no-unused-vars
-  iconForPathPlatform = function(icon: string, size: number) {
-    return ICONS.get(icon) || "icon-file-generic";
-  };
-}
+export const DEFAULT_ICON_SIZE = 16;
 
-
-// eslint-disable-next-line no-magic-numbers
-export function iconForPath(path: string, size = 16) {
+// eslint-disable-next-line no-unused-vars
+export function iconForPath(path: string, size = DEFAULT_ICON_SIZE) {
   const web = /^https?:\/\//.test(path);
   let file = path.split(/[\\/]/).pop();
   if (file) {
@@ -152,7 +144,7 @@ export function iconForPath(path: string, size = 16) {
       file = "file";
     }
   }
-  return iconForPathPlatform(file, size);
+  return ICONS.get(file) || "icon-file-generic";
 }
 
 /**

manifest.json

@@ -1,7 +1,7 @@
 {
   "manifest_version": 2,
   "name": "DownThemAll!",
-  "version": "4.0.5",
+  "version": "4.0.8",
 
   "description": "__MSG_extensionDescription__",
   "homepage_url": "https://downthemall.org/",
@@ -9,6 +9,8 @@
 
   "default_locale": "en",
 
+  "content_security_policy": "script-src 'self'; style-src 'self' 'unsafe-inline'; img-src data: blob: 'self'; connect-src data: blob: 'self'; default-src 'self'",
+
   "icons": {
     "16": "style/icon16.png",
     "32": "style/icon32.png",

style/manager.css

@@ -442,8 +442,13 @@ body > * {
 }
 
 #tooltip-icon {
-  font-size: 48px;
-  line-height: 48px;
+  height: 64px;
+  width: 64px;
+  background-size: 64px 64px;
+  background-repeat: no-repeat;
+  background-position: center center;
+  font-size: 64px;
+  line-height: 64px;
   padding: 6px;
   text-align: center;
   grid-row: 1/-1;

style/single.css

@@ -49,7 +49,7 @@ p.example {
 }
 
 #options > * {
-  margin: 0;
+  margin: 2px;
 }
 
 #options > input {

uikit/css/modal.css

@@ -33,7 +33,7 @@
 .modal-footer {
   display: flex;
   flex-wrap: nowrap;
-  justify-content: right;
+  justify-content: flex-end;
   background: rgba(30, 30, 30, 0.2);
   margin-top: 2em;
   border-top: 1px solid rgba(30, 30, 30, 0.5);

uikit/lib/util.ts

@@ -15,16 +15,20 @@ export function addClass(elem: HTMLElement, ...cls: string[]) {
 
 interface Timer {
   args: any[];
+  id: number;
 }
 
-export function debounce(fn: Function, to: number) {
+export function debounce(fn: Function, to: number, reset?: boolean) {
   let timer: Timer | null;
   return function(...args: any[]) {
     if (timer) {
-      timer.args = args;
-      return;
+      if (!reset) {
+        timer.args = args;
+        return;
+      }
+      window.clearTimeout(timer.id);
     }
-    setTimeout(function() {
+    const id = window.setTimeout(function() {
       if (!timer) {
         return;
       }
@@ -37,7 +41,7 @@ export function debounce(fn: Function, to: number) {
         console.error(ex.toString(), ex);
       }
     }, to);
-    timer = {args};
+    timer = {args, id};
   };
 }
 

util/build.py

@@ -27,6 +27,7 @@ LICENSED = set((".css", ".html", ".js", "*.ts"))
 IGNORED = set((".DS_Store", "Thumbs.db"))
 
 PERM_IGNORED_FX = set(("downloads.shelf",))
+PERM_IGNORED_CHROME = set(("menus",))
 
 SCRIPTS = [
   "yarn build:regexps",
@@ -90,8 +91,6 @@ def build_firefox(args):
   else:
     infos["browser_specific_settings"]["gecko"]["id"] = RELEASE_ID
 
-
-  
   infos["permissions"] = [p for p in infos.get("permissions") if not p in PERM_IGNORED_FX]
   out = Path("web-ext-artifacts") / f"dta-{version}-{args.mode}-fx.zip"
   if not out.parent.exists():
@@ -101,6 +100,33 @@ def build_firefox(args):
   print("Output", out)
   build(out, json.dumps(infos, indent=2).encode("utf-8"))
 
+  
+def build_chrome(args):
+  now = datetime.now().strftime("%Y%m%d%H%M%S")
+  with open("manifest.json") as manip:
+    infos = json.load(manip, object_pairs_hook=OrderedDict)
+
+  version = infos.get("version")
+  if args.mode == "nightly":
+    version = infos["version"] = f"{version}.{now}"
+
+  version = infos.get("version")
+
+  del infos["browser_specific_settings"]
+  if args.mode != "release":
+    infos["version_name"] = f"{version}-{args.mode}"
+    infos["short_name"] = infos.get("name")
+    infos["name"] = f"{infos.get('name')} {args.mode}"
+  
+  infos["permissions"] = [p for p in infos.get("permissions") if not p in PERM_IGNORED_CHROME]
+  out = Path("web-ext-artifacts") / f"dta-{version}-{args.mode}-crx.zip"
+  if not out.parent.exists():
+    out.parent.mkdir()
+  if out.exists():
+    out.unlink()
+  print("Output", out)
+  build(out, json.dumps(infos, indent=2).encode("utf-8"))
+
 def main():
   from argparse import ArgumentParser
   args = ArgumentParser()
@@ -114,6 +140,7 @@ def main():
     else:
       run([script], shell=True)
   build_firefox(args)
+  build_chrome(args)
   print("DONE.")
 
 if __name__ == "__main__":

util/i18ntochrome.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+
+import json
+import re
+from collections import OrderedDict
+from pathlib import Path
+
+re_valid = re.compile("[^A-Za-z0-9_]")
+
+for file in Path("_locales/").glob("**/*.json"):
+  with file.open("r") as filep:
+    messages = json.load(filep, object_pairs_hook=OrderedDict)
+  for x in list(messages):
+    prev = x
+    while True:
+      y = re_valid.sub("_", x)
+      if prev == y:
+        break
+      prev = y
+    if x == y:
+      continue
+    messages[y] = messages[x]
+    del messages[x]
+  with file.open("w", encoding="utf-8") as filep:
+    json.dump(messages, filep, ensure_ascii=False, indent=2)

util/makelocalelist.py

@@ -0,0 +1,16 @@
+#!/usr/bin/env python3
+import json
+from pathlib import Path
+
+langs = sorted(Path("_locales").glob("**/messages.json"), key=lambda p: p.parent.name.casefold())
+all = {}
+for m in langs:
+  loc = m.parent.name
+  with m.open("r") as mp:
+    lang = json.load(mp).get("language").get("message")
+  if not lang:
+    raise Exception(f"{m}: no language")
+  lang = f"{lang} [{loc}]"
+  all[loc] = lang
+with open("_locales/all.json", "wb") as op:
+  op.write(json.dumps(all, indent=2, ensure_ascii=False).encode("utf-8"))

webpack.config.js

@@ -32,7 +32,7 @@ module.exports = {
     if (request === "crypto") {
       return callback(null, "crypto");
     }
-    if (request.includes("_locales")) {
+    if (/_locales.*messages\.json/.test(request)) {
       return callback(null, "null");
     }
     return callback();
@@ -42,6 +42,11 @@ module.exports = {
     filename: "[name].js"
   },
   devtool: "source-map",
+  stats: {
+    hash: true,
+    timings: true,
+    maxModules: 2,
+  },
   watchOptions: {
     ignored: /node_modules|bundles/
   },

windows/icons.ts

@@ -19,7 +19,7 @@ export class Icons extends Map {
     }
     let cls = super.get(url);
     if (!cls) {
-      cls = `icon-${++this.running}`;
+      cls = `iconcache-${++this.running}`;
       const rule = `.${cls} { background-image: url(${url}); }`;
       this.sheet.insertRule(rule);
       super.set(url, cls);

windows/manager.html

@@ -128,8 +128,8 @@
   <template id="menufilter-template">
     <ul>
       <li id="ctx-menufilter-seperator">-</li>
-      <li id="ctx-menufilter-invert" data-autoHide="false">Invert</li>
-      <li id="ctx-menufilter-clear" data-autoHide="false">Clear</li>
+      <li id="ctx-menufilter-invert" data-auto-hide="false">Invert</li>
+      <li id="ctx-menufilter-clear" data-auto-hide="false">Clear</li>
       <li>-</li>
       <li id="ctx-menufilter-sort-ascending" data-icon="icon-sort-asc">Sort ascending</li>
       <li id="ctx-menufilter-sort-descending" data-icon="icon-sort-desc">Sort descending</li>

windows/manager/itemfilters.ts

@@ -340,14 +340,19 @@ export class UrlMenuFilter extends MenuFilter {
   async populate() {
     const filts = await filters();
     for (const i of filts.all.filter(e => e.id !== "deffilter-all")) {
-      this.addItem(i.label, this.toggleRegularFilter.bind(this, i));
+      this.addItem(
+        i.label, this.toggleRegularFilter.bind(this, i), this.filters.has(i));
     }
-    this.addItem("-");
-    sort(
+    const domains = sort(
       Array.from(new Set(this.collection.items.map(e => e.domain))),
       undefined,
       naturalCaseCompare
-    ).forEach(e => {
+    );
+    if (!domains.length) {
+      return;
+    }
+    this.addItem("-");
+    domains.forEach(e => {
       this.addItem(
         e, this.toggleDomainFilter.bind(this, e), this.domains.has(e));
     });
@@ -673,6 +678,11 @@ export class FilteredCollection extends EventEmitter {
       this.emit("sorted");
     }
   }
+
+  invalidateIcons() {
+    this.items.forEach(item => item.clearFontIcons());
+    this.recalculate();
+  }
 }
 
 module.exports = {

windows/manager/table.ts

@@ -33,10 +33,11 @@ import { DownloadState, StateTexts, StateClasses, StateIcons } from "./state";
 import { Tooltip } from "./tooltip";
 import "../../lib/util";
 import { CellTypes } from "../../uikit/lib/constants";
-import { downloads } from "../../lib/browser";
+import { downloads, CHROME } from "../../lib/browser";
 import { $ } from "../winutil";
 // eslint-disable-next-line no-unused-vars
 import { TableConfig } from "../../uikit/lib/config";
+import { IconCache } from "../../lib/iconcache";
 
 const TREE_CONFIG_VERSION = 2;
 const RUNNING_TIMEOUT = 1000;
@@ -52,7 +53,16 @@ const COL_SPEED = 6;
 const COL_MASK = 7;
 const COL_SEGS = 8;
 
+const HIDPI = window.matchMedia &&
+  window.matchMedia("(min-resolution: 2dppx)").matches;
+
 const ICON_BASE_SIZE = 16;
+const ICON_REAL_SIZE = !CHROME && HIDPI ? ICON_BASE_SIZE * 2 : ICON_BASE_SIZE;
+// eslint-disable-next-line no-magic-numbers
+const LARGE_ICON_BASE_SIZE = CHROME ? 32 : 64;
+// eslint-disable-next-line no-magic-numbers
+const MAX_ICON_BASE_SIZE = CHROME ? 32 : 127;
+const LARGE_ICON_REAL_SIZE = HIDPI ? MAX_ICON_BASE_SIZE : LARGE_ICON_BASE_SIZE;
 
 let TEXT_SIZE_UNKNOWM = "unknown";
 let REAL_STATE_TEXTS = Object.freeze(new Map<number, string>());
@@ -108,6 +118,8 @@ export class DownloadItem extends EventEmitter {
 
   public finalName: string;
 
+  public ext?: string;
+
   public position: number;
 
   public filteredPosition: number;
@@ -128,6 +140,10 @@ export class DownloadItem extends EventEmitter {
 
   public mask: string;
 
+  private iconField?: string;
+
+  private largeIconField?: string;
+
   constructor(owner: DownloadTable, raw: any, stats?: Stats) {
     super();
     Object.assign(this, raw);
@@ -138,6 +154,42 @@ export class DownloadItem extends EventEmitter {
     this.lastWritten = 0;
   }
 
+  get icon() {
+    if (this.iconField) {
+      return this.iconField;
+    }
+    this.iconField = this.owner.icons.get(
+      iconForPath(this.finalName, ICON_BASE_SIZE));
+    if (this.ext) {
+      IconCache.get(this.ext, ICON_REAL_SIZE).then(icon => {
+        if (icon) {
+          this.iconField = this.owner.icons.get(icon);
+          if (typeof this.filteredPosition !== undefined) {
+            this.owner.invalidateCell(this.filteredPosition, COL_URL);
+          }
+        }
+      });
+    }
+    return this.iconField || "";
+  }
+
+  get largeIcon() {
+    if (this.largeIconField) {
+      return this.largeIconField;
+    }
+    this.largeIconField = this.owner.icons.get(
+      iconForPath(this.finalName, LARGE_ICON_BASE_SIZE));
+    if (this.ext) {
+      IconCache.get(this.ext, LARGE_ICON_REAL_SIZE).then(icon => {
+        if (icon) {
+          this.largeIconField = this.owner.icons.get(icon);
+        }
+        this.emit("largeIcon");
+      });
+    }
+    return this.largeIconField || "";
+  }
+
   get eta() {
     const {avg} = this.stats;
     if (!this.totalSize || !avg) {
@@ -215,6 +267,9 @@ export class DownloadItem extends EventEmitter {
       PORT.post("all");
       return;
     }
+    if (("ext" in raw) && raw.ext !== this.ext) {
+      this.clearIcons();
+    }
     delete raw.position;
     delete raw.owner;
     const oldState = this.state;
@@ -297,6 +352,20 @@ export class DownloadItem extends EventEmitter {
     this.domain = this.uURL.domain;
     this.emit("url");
   }
+
+  clearIcons() {
+    this.iconField = undefined;
+    this.largeIconField = undefined;
+  }
+
+  clearFontIcons() {
+    if (this.iconField && this.iconField.startsWith("icon-")) {
+      this.iconField = undefined;
+    }
+    if (this.largeIconField && this.largeIconField.startsWith("icon-")) {
+      this.largeIconField = undefined;
+    }
+  }
 }
 
 
@@ -323,7 +392,7 @@ export class DownloadTable extends VirtualTable {
 
   private readonly sids: Map<number, DownloadItem>;
 
-  private readonly icons: Icons;
+  public readonly icons: Icons;
 
   private readonly contextMenu: ContextMenu;
 
@@ -357,6 +426,7 @@ export class DownloadTable extends VirtualTable {
     this.showUrls = new ShowUrlsWatcher(this);
 
     this.updateCounts = debounce(this.updateCounts.bind(this), 100);
+    this.onIconCached = debounce(this.onIconCached.bind(this), 1000);
 
     this.downloads = new FilteredCollection(this);
     this.downloads.on("changed", () => this.updateCounts());
@@ -407,6 +477,8 @@ export class DownloadTable extends VirtualTable {
       col.iconElem.classList.remove("icon-filter");
     });
 
+    IconCache.on("cached", this.onIconCached.bind(this));
+
     this.sids = new Map<number, DownloadItem>();
     this.icons = new Icons($("#icons"));
 
@@ -993,10 +1065,16 @@ export class DownloadTable extends VirtualTable {
         this.updateSizes();
         $("#statusSpeedContainer").classList.remove("hidden");
       }
+      if (item.manId && item.ext) {
+        IconCache.set(item.ext, item.manId).catch(console.error);
+      }
       break;
 
     case DownloadState.DONE:
       this.finished++;
+      if (item.manId && item.ext) {
+        IconCache.set(item.ext, item.manId).catch(console.error);
+      }
       break;
     }
     this.selectionChanged();
@@ -1043,10 +1121,10 @@ export class DownloadTable extends VirtualTable {
     }
     const item = this.downloads.filtered[rowid];
     if (colid === COL_URL) {
-      return this.icons.get(iconForPath(item.finalName, ICON_BASE_SIZE));
+      return item.icon;
     }
     if (colid === COL_PROGRESS) {
-      return StateIcons.get(item.state);
+      return StateIcons.get(item.state) || null;
     }
     return null;
   }
@@ -1119,4 +1197,8 @@ export class DownloadTable extends VirtualTable {
       return -1;
     }
   }
+
+  onIconCached() {
+    this.downloads.invalidateIcons();
+  }
 }

windows/manager/tooltip.ts

@@ -146,6 +146,8 @@ export class Tooltip {
   constructor(item: DownloadItem, pos: number) {
     this.update = this.update.bind(this);
     this.item = item;
+    this.item.on("largeIcon", this.update);
+
     const tmpl = (
       document.querySelector<HTMLTemplateElement>("#tooltip-template"));
     if (!tmpl) {
@@ -178,7 +180,7 @@ export class Tooltip {
       this.dismiss();
       return;
     }
-    const icon = item.owner.getCellIcon(item.filteredPosition, 0);
+    const icon = item.largeIcon;
     this.icon.className = icon;
     this.name.textContent = item.destFull;
     this.from.textContent = item.usable;
@@ -318,5 +320,6 @@ export class Tooltip {
     }
     this.item.off("stats", this.update);
     this.item.off("update", this.update);
+    this.item.off("largeIcon", this.update);
   }
 }

windows/popup.html

@@ -7,6 +7,12 @@
       box-sizing: content-box !important;
     }
 
+    html, body {
+      height: auto !important;
+      -webkit-user-select: none;
+      user-select: none;
+    }
+
     ul {
       margin: 1.5ex;
       margin-right: 2ex;
@@ -25,6 +31,8 @@
       vertical-align: center;
       align-items: center;
       border-radius: 4px;
+      cursor: default;
+      white-space: nowrap;
     }
 
     li.sep {

windows/popup.ts

@@ -6,7 +6,9 @@ import { localize } from "../lib/i18n";
 declare let browser: any;
 declare let chrome: any;
 
-const runtime = browser !== "undefined" ? browser.runtime : chrome.runtime;
+const runtime = typeof browser !== "undefined" ?
+  browser.runtime :
+  chrome.runtime;
 
 function handler(e: Event) {
   e.preventDefault();

windows/prefs.html

@@ -72,6 +72,12 @@
     </fieldset>
     <fieldset>
       <legend>Translations</legend>
+      <label>Language:
+        <select id="languages">
+          <option value="default">Browser default</option>
+        </select>
+      </label>
+      <hr>
       <div>
         <button id="loadCustomLocale">Load custom translation</button>
         <button id="clearCustomLocale">Clear custom translation</button>

windows/prefs.ts

@@ -1,7 +1,13 @@
 "use strict";
 // License: MIT
 
-import { _, localize, saveCustomLocale } from "../lib/i18n";
+import {
+  ALL_LANGS,
+  _,
+  getCurrentLanguage,
+  localize,
+  saveCustomLocale,
+} from "../lib/i18n";
 import { Prefs, PrefWatcher } from "../lib/prefs";
 import { hostToDomain } from "../lib/util";
 import { filters } from "../lib/filters";
@@ -13,7 +19,7 @@ import { iconForPath, visible } from "../lib/windowutils";
 import { VirtualTable } from "../uikit/lib/table";
 import { Icons } from "./icons";
 import { $ } from "./winutil";
-import { runtime } from "../lib/browser";
+import { runtime, storage } from "../lib/browser";
 
 const ICON_BASE_SIZE = 16;
 
@@ -544,8 +550,8 @@ class LimitsUI extends VirtualTable {
 }
 
 
-addEventListener("DOMContentLoaded", () => {
-  localize(document.documentElement);
+addEventListener("DOMContentLoaded", async () => {
+  await localize(document.documentElement);
 
   // General
   new BoolPref("pref-global-turbo", "global-turbo");
@@ -557,7 +563,7 @@ addEventListener("DOMContentLoaded", () => {
   new BoolPref("pref-text-links", "text-links");
   new BoolPref("pref-add-paused", "add-paused");
   new BoolPref("pref-show-urls", "show-urls");
-  new BoolPref("pref-remove-missing-on-init", "show-urls");
+  new BoolPref("pref-remove-missing-on-init", "remove-missing-on-init");
   new OptionPref("pref-conflict-action", "conflict-action");
 
   $("#reset-confirmations").addEventListener("click", async () => {
@@ -587,6 +593,29 @@ addEventListener("DOMContentLoaded", () => {
       _("information.title"), _("reset-layouts.done"), _("ok"));
   });
 
+
+  const langs = $<HTMLSelectElement>("#languages");
+  const currentLang = getCurrentLanguage();
+  for (const [code, lang] of ALL_LANGS.entries()) {
+    const langEl = document.createElement("option");
+    langEl.textContent = lang;
+    langEl.value = code;
+    if (code === currentLang) {
+      langEl.selected = true;
+    }
+    langs.appendChild(langEl);
+  }
+  langs.addEventListener("change", async () => {
+    await storage.sync.set({language: langs.value});
+    if (langs.value === currentLang) {
+      return;
+    }
+    // eslint-disable-next-line max-len
+    if (confirm("Changing the selected translation requires restarting the extension.\nDo you want to restart the extension now?")) {
+      runtime.reload();
+    }
+  });
+
   // Filters
   visible("#filters").then(() => new FiltersUI());
 
@@ -621,7 +650,7 @@ addEventListener("DOMContentLoaded", () => {
         reader.readAsText(file);
       });
       saveCustomLocale(text);
-      if (confirm("Imported your file.\nWant to realod the extension now?")) {
+      if (confirm("Imported your file.\nWant to relaod the extension now?")) {
         runtime.reload();
       }
     }