gowiki

Gowiki is an example of a very tiny web application that is already useful: it can show text files (GET /view/<PAGE>) from ./pages/, edit them using a <textarea> (GET /edit/<page>) and save them back (POST /save/<PAGE>).

http.server: the basics

First of all, we need a web server. I use python3 -m http.server pretty often to serve a directory over http quickly, which looks promising to extend.

The documentation page for http.server greets us with a warning:

http.server is meant for demo purposes and does not implement the stringent security checks needed of real HTTP server. We do not recommend using this module directly in production.

Fine, that must be one of the reasons why nobody uses it. Let’s proceed at our own risk.

Let’s run the initial example:

import http.server as http

address = ('', 8000)
with http.HTTPServer(address, SimpleHTTPRequestHandler) as httpd:
    print('Listening on %s:%d' % address)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass

Extending the http.server

The documentation looked pretty massive and intimidating at first, and after some initial reading I had a lot of questions:

• should I extend HTTPServer, BaseHTTPRequestHandler, one of its subclasses? Turns out that HTTPServer is the “transport” part of the application (a subclass of TCPServer) and it takes a request handler (subclassed from BaseHTTPRequestHandler) as a mix-in to actually handle the requests; SimpleHTTPRequestHandler is a subclass of BaseHTTPRequestHandler that maps requests to files under the current directory; CGIHTTPRequestHandler extends SimpleHTTPRequestHandler with the ability to run CGI scripts from ./cgi-bin (in the very primitive CGI way: running a process for each request).
• are there any request routing mechanism in the standard library? Nope, none: you need to handle routing manually, unlike Go;
• where do I get the requested path (do I get any parameters to do_GET)? I needed to check with the source code and debug it to understand that do_GET relies on the state of its object;
• what is send_response(), send_response_only(), send_error(), end_headers(), what is the protocol to call them? Which of those are internal methods and which can I call? I needed to read the source code of http.server to understand what they do exactly;
• how do I write the response? I wrote some Python FastCGI script some time ago, so I knew about wfile file descriptors that can be used for this: indeed, there’s self.rfileand self.wfile;

import shutil
from pathlib import Path
from http import HTTPStatus as status

pages_dir = Path('./pages/')

class RequestHandlers(http.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path.startswith('/view/'):
            return self.get_view()
        return self.send_error(status.NOT_FOUND)

    def respond_ok(self, body):
        self.send_response(status.OK)
        self.end_headers()

        shutil.copyfileobj(body, self.wfile)

    def get_view(self):
        title = self.path[len('/view/'):]

        p = pages_dir / (title + '.txt')
        if not p.exists():
            return self.send_error(status.NOT_FOUND)

        with p.open('rb') as f:
            self.respond_ok(f)

Plug it in http.HTTPServer(address, RequestHandlers) and we are ready to go.

Serving HTML

Now let’s serve html instead of plain text. At first, I had hard-coded html snippets in the source code using multi-line string literals and inserting values using formatting, but soon I needed to move templates into separate files and to do more advanced templating using string.Template:

...
import io
import string

templ_dir = Path('./tmpl/')

class RequestHandlers(http.BaseHTTPRequestHandler):
#   ...
    def respond_ok(self, body, mime='plain'):
        if isinstance(body, str):
            body = body.encode('utf8')
        if isinstance(body, bytes):
            body = io.BytesIO(body)

        self.send_response(status.OK)
        self.send_header('Content-Type', 'text/%s; charset=utf8' % mime)
        self.end_headers()
        
        shutil.copyfileobj(body, self.wfile)

    def format_tmpl(self, page, **values):
        template = open(templ_dir / tmpl).read()
        t = string.Template(template)
        return t.substitute(**values)

    def get_view(self):
        title = self.path[len('/view/'):]

        p = pages_dir / (title + '.txt')
        if not p.exists():
            return self.send_error(status.NOT_FOUND)

        with p.open('rb') as f:
            body = f.read().decode('utf8')
            page = self.format_tmpl('view.html', title=title, body=body)

            self.respond_ok(page, text='html')

and now we have a new file, tmpl/view.html:

<h1>${title}</h1>
<ul id="menu">
<li>[<a href="/">main</a>]</li>
<li>[<a href="/edit/${title}">edit</a>]</li>
</ul>
<hr>
<div><pre>${body}</pre></div>
<link rel="stylesheet" href="/file/style.css">

Wow, that’s a rather low-level trickery going on now in respond_ok(), but it will pay off later, when we need to serve files.

Editing…

The frontend part is a matter of a simple html page, tmpl/edit.html:

<h1>${title}<h1>
<ul id="menu"></ul>
<hr>
<div>
<form action="/save/${title}" method="POST">
<textarea cols="80" rows="20" autofocus name="body">${body}</textarea><br>
<input type="submit" value="Save">
</form>
</div>

and serving it is easy:

class RequestHandlers(http.BaseHTTPRequestHandler):
#   ...
    def do_GET(self):
        if self.path.startswith('/view/'):
            return self.get_view()
        if self.path.startswith('/edit/'):
            return self.get_edit()
        return self.send_error(status.NOT_FOUND)

    def get_edit(self):
        title = self.path[len("/edit/"):]

        body = ''
        p = pages_dir / (title + '.txt')
        if p.exists():
            body = p.open().read()

        page = self.format_tmpl('edit.html', title=title, body=body)

        self.respond_ok(page, text='html')

…and POSTing

Now, how do I do POST requests? The shocking answer is that http.server only provides do_POST and leaves you on your own. You need to read the sent form yourself, parse it (thanks to the standard library, there’s urllib.parse.parse_qs) and save it.

I spent some time debugging why self.rfile.read() just hangs the app: turns out, you need to read Content-Length and only read it from self.rfile (HTTP/1.1 can reuse connections for new requests).

class RequestHandlers(http.BaseHTTPRequestHandler):
#   ...
    def _is_form(self):
        """ checks is the request is a form sent from a page """
        formmime = 'application/x-www-form-urlencoded'
        return self.headers.get('content-type') == formmime

    def _parse_form(self):
        """ parse and save form into `self.form` """
        size = int(self.headers['content-length'])
        form = self.rfile.read(size)
        form = form.decode('ascii')
        form = parse_qs(form)
        self.form = form

    def form_value(self, key):
        """ get a form parameter by key or None """
        return self.form.get(key, [None])[0]

– and now we are ready to read form parameters!

    def do_POST(self):
        if self._is_form():
            self._parse_form()

        if self.path.startswith('/save/'):
            return self.post_save()

        return self.send_error(status.NOT_FOUND)

    def post_save(self):
        title = self.path[len("/save/"):]
        body = self.form_value('body')

        p = pages_dir / (title + '.txt')
        with p.open('w') as f:
            f.write(body)

        self.respond_redirect('/view/' + title)

Redirecting requires reinventing the wheel again:

    def respond_redirect(self, url):
        self.send_response(status.FOUND)
        self.send_header('Location', url)
        self.end_headers()

Now we can view, create and edit pages.

Main page

Adding the index page is very straightforward now. The only small problem is that there are no loops in string.Template, so the html representation for the list of pages must be built in Python:

    def get_main(self):
        pages = []
        for p in pages_dir.iterdir():
            if p.suffix == '.txt':
                pages.append(p.stem)

        pagelist = []
        for page in pages:
            pagelist.append(f'<li><a href="/view/{page}">{page}</a></li>')
        pagelist = '\n'.join(pagelist)

        body = self.format_tmpl('main.html', pagelist=pagelist)
        self.respond_ok(body, text='html')

    def post_goto(self):
        page = self.form_value('page')
        self.respond_redirect("/edit/" + page)

– and don’t forget to add get_main to do_GET and post_goto to do_POST! This definitely could be automated using getattr and dynamic method calls, but I am a little wary of calling code dynamically based on external requests.

main.html is:

<h1>pywiki</h1>
<hr>
<form action="/goto" method="POST">
<input type="text" name="page" required autofocus>
<input type="submit" value="Go">
</form>
<ul>
${pagelist}
</ul>

Serving files

What about serving files, e.g. adding a bit of CSS through /file/style.css? I could just fall back onto SimpleHTTPRequestHandler and use its do_GET, but let’s reinvent this too:

files_dir = Path('./file/')

class RequestHandlers(http.BaseHTTPRequestHandler):
#   ...

    def get_file(self):
        """ just serve files from ./file/ """
        fname = self.path[len("/file/"):]
        if '/' in fname:
            return self.send_error(status.BAD_REQUEST)

        fpath = files_dir / fname
        if not fpath.exists():
            return self.send_error(status.NOT_FOUND)

        with fpath.open('rb') as f:
            contenttype = 'application/octet-stream'
            if fname.endswith('.css'):
                contenttype = 'text/css'

            self.respond_ok(f, {'Content-Type': contenttype})

A nice side-effect here is that it’s not necessary to read file to memory at all, shutil.copyfileobj() can take care of this through sendfile(2).

General ergonomics compared to Go

• absence of any complex examples in the documentation. This might be intentional: to discourage production usage, which is assumed by default in Go.
• massive documentation with several big classes with directly accessible state; you need to read through and consult the source code to understand what is going on: what methods to call, in what sequence, what methods are low-level and high-level, it was not clear immediately how to read the request body and write the response (which is immediately obvious in Go: http.Request and http.ResponseWriter as parameters to a handler);
• not a very complex, but non-trivial hierarchy of classes which are used to access complex state and run custom methods (instead of clean and isolated callbacks in Go with clear inputs and outputs);
• a very bare HTTP implementation, without essentials like parsing form fields (Go’s http.Request.FormValue()) and redirects (just http.Redirect() in Go);
• no html templating with loops, no html character escaping, compared to http/template in Go;

Conclusion

The source code of pywiki may be found here: https://git.dmytrish.net/lang-learn/pywiki.

I feel like I have written my own buggy and incomplete micro webframework at this point: low-level http manipulation, absence of good url routing, manual parsing, etc; http.server is clearly not supposed to be extended further and should be a finished demo of what can be done using other parts of the standard library.

There is no concurrency here: everything is blocking (although e.g. Flask works the same way); even though Python has asyncio now, the standard library does not ship an asyncio http server, which is a shame. Go gives you production-grade concurrency for free.

I haven’t built any security checks into my code, which might be disastrous.

So: you can write primitive web-applications using only the Python standard library. Should you do it? Judge for yourself.

I consider the core of the implementation reasonably self-sufficient by now: lambdas and macros work reasonably well (with proper dynamic scoping and Lisp-2 namespace separation, unlike Samsonjs’s elisp.js from 2009!), code may be evaled, the most used types are implemented (symbols, lists, strings, numbers, bools, vectors).

On the other hand, the implementation is still very bare, lacks many essential types (e.g. hashtables) and even language-level functionality (e.g. &optional and &rest arguments in functions), still wants for proper error handling and debugging and it completely lacks any chrome around the core primitives (even defmacro and defun are very recent additions).

Also, my implementation is not a transpiler in the strict sense of “code in, code out”: Lisp is still too dynamic and fluid in runtime to map into some static Javascript. Macros make a full transpiler necessary even in runtime. I’d rather call it a “Just-In-Time Transpiler”.

If you are still keen to try it and are fine with using its source code as a reference, you can find a basic online demo here: https://dmytrish.net/elisp.

Why?

My first and the most moon-shot reason has been the dream of seeing org-mode in the browser. I still hope that I might be able to write just enough functionality to support significant chunk of org-mode on top of, say, Ymacs.

Also, I’ve been curious to learn more about Javascript/v8/node.js and v8’s performance. This goal has definitely paid off: I gained some insight into speeding up Javascript code and inner workings of v8, even though I am still very far from an expert in this.

Development notes

Test-Driven Development is cool–when you have a clear specification/reference implementation like the Elisp reference manual and Emacs (just look at this for example!). The right amount of end-to-end tests makes writing Javascript almost not scary, Mocha is pretty fast and provides with nice guardrails to structure lots of tests. Hopefully, extensive testing will make the project at least a bit more maintainable and evolvable.

Dynamic scoping. With lexical scoping (i.e. almost any other kind of Lisp) translation to Javascript would be a lot easier (at least leaving out macros), but Elisp is the only surviving dinosaur of dynamic scoping, so I had to map dynamic bindings into Javascript carefully. This hinders v8 from many optimizations (obscuring the intent) and produces ugly JS code like this:

elisp> (defun sqr (x) (* x x))
(lambda (x) (* x x))

elisp> (jscode (symbol-function 'sqr))
"(() => {
  let f1 = global.env.fun('*');
  let v2 = global.env.var_('x');
  return f1.get().fcall([v2.get(), v2.get()], global.env);
})"

Pain of macros. Before I started working on macros in Elisp, the source code organization was straightforward and dependencies were unidirectional: I had types.js describing Lisp types, parser.js was turning an input string to a Lisp object, translate.js was taking a Lisp object and producing ephemeral JS code from it; elisp.js nicely packed everything into a high-level wrapper.

Once I’d added macros, the architecture became a mess: now even types.js requires elisp.eval_lisp(), creating circular references and lazy-loading in parser.js and translate.js. Maybe, I will be able to abstract the runtime parts from these files later, but macros really made the interpreter source code tightly coupled and intertwined. Just look at this pearl (yes, every function call might require a recompilation in Elisp):

LispFun.prototype.fcall = function(args, env) {
  try {
    return elisp.fcall.call(this, args, env);
  } catch (e) {
    if (e.message !== 'Macro accessed as a function')
      throw e;
    /* trigger recompilation */
    this.func = undefined;
    return elisp.fcall.call(this, args, env);
  }
};

Repl tab completion is easy and surprisingly useful. My heuristic for tab completion is very crude and dumb at the moment: just looking back until the start of the current symbol and if there’s a (, complete the symbol as a function, otherwise complete it as a variable. Still, it works pretty reliably and saves a lot of typing. Also, this gave me a little insight why on earth anybody would prefer Lisp-2 (an ugly, ugly idea compared to Lisp-1!): now I understand that Lisp-2 might be slightly easier for tooling.

Profiling and benchmarking

v8 is fast!

Still, my first naive version that could run silly-loops made even v8 stumble compared to the slow Emacs interpreter. I had to learn about v8 profiling and it took several significant optimizations for it to become competitive with Emacs (this shows execution time in seconds):

$ npm run bench

> elisp.js@0.1.0 bench /home/mitra/code/elisp.js
> ./bench/bench

funcall
     now: 0,285
   emacs: 0,255
silly-loop
     now: 0,834
   emacs: 1,845

There is no free lunch, a slow implementation is still a slow implementation even on a fast engine. Profiling can guide to faster code though, so it’s better to say “v8 can be fast”.

Still, I find the results pretty satisfying for my limited resources.

What exactly is done in 0.1.0

Citing the CHANGELOG,

• language types: integer, symbol, nil, cons, string, vector; LispFun and LispMacro as invisible “upgrades” to lists when appropriate;
• language special forms: let, lambda, quote, setq, if, progn, while;
• language subrs (builtin functions): subrp, functionp, macrop, listp, numberp, booleanp, read, eval, macroexpand-1, jscode, jseval, +, -, *, <, car, cdr, cons, list, fset, symbol-function, error, print, float-time.
• functions: positional arguments only, dynamic scoping;
• UI: repl.js with defmacro, defun, defvar; tab-completion of functions and variables;
• UI: a browser demo using webpack;
• benchmarked and optimized, in the ballpark of Emacs interpreter or faster.

Things to do before 0.2.0

The main goals for 0.2.0 at the moment are:

• Make it parse and interpret non-trivial amount of source code from lisp/emacs-lisp/ of Emacs;
• Add meaningful robust error handling with stacktraces, pluggable debuggers and references to source code;
• Extend the standard library of subrs on the go as needed;
• Give more thought to Emacs-specific types like buffers, cursors, charmaps, etc;
• Starting at least some form of documentation.

Still, a small problem needed a fix: the input field did not focus automatically and I had to use touchpad to click it, which is quite distracting. My ideal dictionary applet workflow is: reading a text, spotting an unknown/unclear word, pressing Shift+Win+D, typing the word in (typing is good for remembering!), getting results on Enter, Esc, get back to the text.

Hey, it’s the Linux tinkering time again!

Disclaimer: I am not an expert in KDE by any means, take the recipes with a grain of salt; if you are, please point out what can be done better.

Finding the source for the widget

I worked with Qt a little, a long time ago, so I knew about QML: it’s a fun little declarative language on top of Qt used to describe interfaces and UI actions. Turns out, you can write whole widgets in it!

Let’s find the source of the dictionary widget then.

After a few guesses (plasmoid? widget? dict?), dpkg --list | grep (I am using Kubuntu) gave me this:

$ dpkg --list | grep plasma
...
ii  plasma-widgets-addons   4:5.12.6-0ubuntu0.1    amd64    additional widgets for Plasma 5
...

Looks like the dictionary applet might be in there:

$ dpkg -L plasma-widgets-addons | grep dict
...
/usr/share/plasma/plasmoids/org.kde.plasma_applet_dict
...

Right. Examining the source showed that the applet UI is described here: /usr/share/plasma/plasmoids/org.kde.plasma_dict_applet/contents/ui/main.qml.

Looking for the focus event

Now it was time to actually fix the problem: get the text field (input) to focus every time the plasmoid is shown.

First, I tried to do it the right way: reading the documentation, understanding QML events and layouts, etc.

Googling “QML widget events” led me to the official QML documentation, which looks pretty solid and systematic. Sadly, I had spent a couple of hours trying to find anything relevant in the dump of definitions of methods and properties (scattered over super- and supersuperclasses of TextField and ColumnLayout), trying to grep through tutorials and StackOverflow – before giving up.

Then I had a better idea: installing and running QtCreator, opening the main.qml file and just stupidly trying a few completions here and there. This proved to be surprisingly effective and trying a few on completions found the onActiveFocusChanged callback and .focus property of the input element:

ColumnLayout {
    onActiveFocusChanged: input.focus = true
    
    ...
}

Now, let’s test the changes:

$ pkill plasmashell ; kstart5 plasmashell

Nice, Shift+Win+D now showed the dictionary widget with the search field focused.

A note: it’s better to keep the modified main.qml in a safe place and copy it in /usr/share/..., because it might be rewritten by a new system update. I omitted copying it for simplicity.

Making the previous search selected

Another annoying usability problem remained: now the search field was focused, but contained the previous search in it, unselected, so I had to be careful: if there was a word in the search field, I had to erase it first (Shift+Backspace) or I would just append a new word to its end. It was not a big deal, but it introduced some state I needed to track every time and any state is distracting (Vim is still a holy cow that is above any whining about modes)!

The problem could be solved by clearing the field every time the widget was hidden, but sometimes you just want to see the previous word, not to look up a new one. Selecting the previous search when the widget is shown solves this: the previous search is here, but typing anything replaces it.

This time the documentation was more helpful:

ColumnLayout {
    onActiveFocusChanged: {
        input.selectAll()
        input.focus = true
    }
    
    ...
}

Linux is still DIY (Design It Yourself) sometimes!

While we’re at it, why not fix the annoying font and background color choice with my crazy design skills? I find it annoying and not as streamlined as the Mac widget, don’t you (let the flame war begin!)?

Trying to tinker with QML Font property was not successful. WebEngineView is obviously a piece of web that listens only to CSS. Quick googling shows that there is no easy way to style WebEngineView other than running some Javascript (I will be glad to be proved wrong here!):

ColumnLayout {
    ...
    WebEngineView: {
        id: web
        ...
        onLoadingChanged: {
            web.runJavaScript(
                "document.body.style['font-family'] = 'Lato, monospace'; " +
                "document.body.style['line-height'] = '1.5'; " +
                "document.body.style['background-color'] = '#f9f7e0'; "
            )
        }
    }
}

Now it was much more palatable to me:

Conclusions

• The KDE dictionary widget is very basic, quick and dirty, and could use some love and usability testing. On the other hand, I am satisfied with my tweaks and resulting usability (should I contribute that upstream?).

• KDE widgets are small, fun and flexible (for a programmer) and fixing them does not require to recompile half of KDE.

• QtCreator was more helpful than the docs, the QML documentation is a OOP mess.

I learned this in the abyss of info bash: it mentions that redirects like /dev/stdin, /dev/fd/FD, etc, are treated specially. One of redirections on the list was /dev/tcp/HOST/PORT—I could not believe my eyes!

After figuring out how to use the same file descriptor for reading and writing, I was able to make a simple HTTP request to Google:

$ (exec 3<>/dev/tcp/www.google.com/80 ; 
echo "GET /search?q=bash+madness HTTP/1.0" >&3; 
echo >&3; 
cat <&3 ) | less

Yes, it’s that simple.

I am greatly satisfied with my non-orthodox KDE setup with a few tweaks, except for one little thing I can’t even properly google (I get a lot of results for keyboard customization).

I noticed a weird thing: MacOS never gets multi-modifier shortcuts (Ctrl-Shift, Alt-Cmd, etc) wrong: I never get Ctrl-D/Shift-D instead of Ctrl-Shift-D, which happens pretty often on my new Linux system.

This could be explained by the hypothesis that the MacOS input layer has a special delay to account for this: when Ctrl, D are pressed in a short time, it waits an instant for another modifier key to follow (e.g. Shift) and produces Ctrl-Shift-D. The XWindow (xinput?) behavior seems more straightforward: it generates an event for Ctrl-D the instant when D is down, disregarding any modifier keys after that.

It’s a small thing, but quite annoying already.

It was a nice spring evening when this happened. At 19:41, I came home, pulled my phone out of my pocket and saw this:

For some surprising and possibly alarming reason, the official Telegram app for Android needed access to camera while in the background! Was it Telegram itself watching me for no reason, was it a bug in the app, was it a bug in the Android permission monitoring?

I was perfectly sure that I did not use the phone at the time (I knew the exact minute I used it last time before this happened: a picture was taken at 19:25) and that Telegram was in the background.

I took the screenshot above and started thinking about the situation and what I can do to figure this out. Good ideas come slowly, and when I tried to get the phone logs with adb logcat, it was already too late: logs for that moment were gone.

So, I noted that I would need to logcat the logs later when a suspicious activity happens again and waited.

Next Monday:
Telegram blinks with the camera torchlight when opened

Next time this happened in the morning when I was checking my phone: this time Telegram blinked with the frontal camera right after I opened it. I was ready this time and pulled full logs from the phone.

Here it was, caught with a smoking gun:

04-30 12:10:01.032   429 20402 I CameraService: CameraService::connect call (PID -1 "org.telegram.messenger", camera ID 1) for HAL version default and Camera API version 1

This proved that Telegram used my cameras on its own, without asking/warning me, and this time it appeared smarter than before: it did not try accessing a camera from background this time, it waited until I opened the app instead.

Why the app would do that? Digging further revealed that Telegram accessed both rear and frontal camera for a moment, changing torchlight status (this is a clean excerpt, the full log is very verbose and often low-level):

04-30 12:10:01.738   429 20402 I CameraService: torch_mode_status_change: cameraId=1, TorchStatus=1
04-30 12:10:01.738   429 20402 I CameraService: onTorchStatusChangedLocked: Torch status changed for cameraId=1, newTorchStatus=1
04-30 12:10:01.739  2081 16047 I CameraManagerGlobal: postSingleTorchUpdate : Camera id 1 has torch status changed to 0x1
04-30 12:10:01.739   701  6521 I CameraManagerGlobal: postSingleTorchUpdate : Camera id 1 has torch status changed to 0x1
04-30 12:10:01.741   429   740 W QCamera2Factory: getCameraInfo: E, camera_id = 1
04-30 12:10:01.741   429   740 I QCamera2HWI: getFlashInfo cameraId:0 hasFlash:1
04-30 12:10:01.741   429   740 I QCamera2HWI: getCapabilities id:0 hasFlash:1
04-30 12:10:01.741   429   740 W QCamera2HWI: __dbg: info->orientation : 90 rc = 0
04-30 12:10:01.741   429   740 I QCamera2HWI: getFlashInfo cameraId:1 hasFlash:1
04-30 12:10:01.741   429   740 I QCamera2HWI: getCapabilities id:1 hasFlash:1
04-30 12:10:01.741   429   740 W QCamera2HWI: __dbg: info->orientation : 270 rc = 0
04-30 12:10:01.741   429   740 W QCamera2Factory: getCameraInfo: X
04-30 12:10:01.744   429 20402 I CameraService: CameraService::connect call (PID -1 "org.telegram.messenger", camera ID 1) for HAL version default and Camera API version 1

04-30 12:10:01.750   429 20402 E CameraService: checkTemperature (0)
04-30 12:10:01.750   429 20402 I CameraService: CameraService::validateConnectLocked is ok : calling pid 30732, calling uid 10137, cameraservice pid=429, device user 0, currently allowed device users: 0
04-30 12:10:01.750   429 20402 I CameraFlashlight: prepareDeviceOpen: prepare for device open(cameraId=1, package=org.telegram.messenger)
04-30 12:10:01.750   429 20402 I CameraService: torch_mode_status_change: cameraId=0, TorchStatus=0
04-30 12:10:01.750   429 20402 I CameraService: onTorchStatusChangedLocked: Torch status changed for cameraId=0, newTorchStatus=0
04-30 12:10:01.750   429 20402 I CameraService: torch_mode_status_change: cameraId=1, TorchStatus=0
04-30 12:10:01.750   429 20402 I CameraService: onTorchStatusChangedLocked: Torch status changed for cameraId=1, newTorchStatus=0
04-30 12:10:01.751   429 20402 W QCamera2Factory: getCameraInfo: E, camera_id = 1
04-30 12:10:01.751   429 20402 I QCamera2HWI: getFlashInfo cameraId:0 hasFlash:1
04-30 12:10:01.752   429 20402 I QCamera2HWI: getCapabilities id:0 hasFlash:1
04-30 12:10:01.752   429 20402 W QCamera2HWI: __dbg: info->orientation : 90 rc = 0
04-30 12:10:01.752   429 20402 I QCamera2HWI: getFlashInfo cameraId:1 hasFlash:1
04-30 12:10:01.752   429 20402 I QCamera2HWI: getCapabilities id:1 hasFlash:1
04-30 12:10:01.752   429 20402 W QCamera2HWI: __dbg: info->orientation : 270 rc = 0
04-30 12:10:01.752   429 20402 W QCamera2Factory: getCameraInfo: X

04-30 12:10:01.765   429 20402 I CameraService: torch_mode_status_change: cameraId=1, TorchStatus=0
04-30 12:10:01.765   429 20402 I CameraService: onTorchStatusChangedLocked: Torch status changed for cameraId=1, newTorchStatus=0

This log is cherry-picked and omits activation of AudioSink, MediaPlayerServer, OggExtractor, and lots of other low-level things, but I suppose that this activity is automatic when opening a camera; I could not find any explicit signs of a picture/video being taken, please feel free to examine the full log and tell me I am wrong.

Conclusion

It seems rather innocent: as far as I see, this looks like a instantaneous activation of camera torchlight without clear signs of a picture being taken. It does not look like a sneaky operation: it blinked the camera light, obviously drawing my attention. On the other hand, I do not have extensive knowledge of low-level details of camera operation on Android and I could have missed something.

The most probable benign explanation sounds like this: Telegram has been experimenting with blinking camera torchlight on new messages for phones that do not have a LED above the screen dedicated to notifications (like in many Sony smartphones). The first time they were releasing this feature, Permission Manager was raising warnings about the app using camera from the background. This explains the first time: I was coming home, my phone connected to home Wi-Fi from the street, Telegram detected new messages and tried to blink about this with the torchlight from the background. The second time they fixed the background access and restricted blinking to foreground mode.

Still, I do not feel comfortable with Telegram accessing my camera without my intent. The blinking feature (as I guess it) sounds dubious and unnecessary to me (I try to restrict notifications on my phone as much as I can) and it’s easy to sneak any malicious activity later, covering camera activity with innocent explanations.

#include <stdio.h>
 
#include <map>
#include <string>
 
int main() {
  std::map<std::string, std::string> x;
  int i = 0x5001;
  x["foo"] = i;
 
  printf("%s", x["foo"].c_str());
 
  return 0;
}

So, what is happening here? Compilation does not give any warnings and it runs without segfaults, and more, without any visible output!

What happened to type safety?! How on earth is this possible?! Is it a compiler bug? No, it turns out that it’s perfectly valid C++ (C++98, to be pedantic!).

What is x[“foo”], exactly?

My first reflex was to look at the assembly:

$ clang++ -Wall -Wextra -g strmap.cc
$ otool -tv ./a.out | c++filt

and oops, trying to wade through C++ heavily templated symbol names was an exercise in futility (unlike plain C).

Ok, let’s fire up a debugger, which is by definition more human-friendly, and set a breakpoint after the mysterious assignment statement and see what we got:

$ lldb ./a.out
Current executable set to './a.out' (x86_64).
(lldb) b 11
Breakpoint 1: where = a.out`main + 354 at strmap.cc:11, address = 0x0000000100000bc2
(lldb) r
Process 25081 launched: './a.out' (x86_64)
Process 25081 stopped
* thread #1: tid = 0x45b804, 0x0000000100000bc2 a.out`main + 354 at strmap.cc:11, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100000bc2 a.out`main + 354 at strmap.cc:11
   8      int i = 0x5001;
   9      x["foo"] = i;
   10    
-> 11     printf("%s", x["foo"].c_str());
   12    
   13     return 0;
   14   }
(lldb) p x
(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<const std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >) $0 = size=1 {
  [0] = (first = "foo", second = "\x01")
}
(lldb) 

Whoa, the string "\x01" is inserted silently! Is this some kind of integer-pointer conversion that creates a C string from the integer? If it is, why there are no segfaults (0x5001 does not look like a memory address usually used in Unix-like OSes). If it a pointer conversion, let’s try to produce a segfault by changing 0x5001 to something else, like 0x6001. Still no segfaults, and the string is still "\x01". So, it’s not a pointer, it’s the integer value that is changed into "\x01" somehow. Now it’s clear that int is silently converted to char, leaving us with the least significant byte.

Let’s make the source code a little less obscure:

x["foo"] = 'a';

String constructors?

Now we have a new mystery: how can you put a char into map<string, string>?

The usual suspects in such investigations are usually constructors that can kick in silently and make everything into everything. Is it a std::string constructor that would take a char (business as usual in the C++ land, why not)?

Let’s read the docs on basic_string constructors. Still, there is no sign of a constructor that would take a bare CharT value as an argument.

Is it an initializer_list somewhere that could be made with one char? No, clang++ -std=c++98 strmap.cc is happy to produce the same result.

Another twenty minutes are fruitlessly spent on trying to guess what other constructor could be hidden here, until an experiment proves me wrong:

x["foo"] = std::string('a');

spews two screens of compilation errors that can be summarized as “no such constructor”.

Who else was there?

Now the investigation turns to every witness of the crime, one of which is std::map. How does std::map::operator[] work? Reading the docs clears this up a bit:

1) Inserts value_type(key, T()) if the key does not exist. This function is equivalent to return insert(std::make_pair(key, T())).first->second;

• key_type must meet the requirements of CopyConstructible.
• mapped_type must meet the requirements of CopyConstructible and DefaultConstructible.

If an insertion is performed, the mapped value is value-initialized (default-constructed for class types, zero-initialized otherwise) and a reference to it is returned.

So that’s how the string "\x01" is created: it emerges as an empty string in x["foo"] and gets its content later. Now it’s clear that the default constructor is called and the content is assigned later through std::string::operator=.

std::string::operator= in its fourth (sic) form is the last piece of the puzzle:

4) Replaces the contents with character ch as if by *this = basic_string(1,c)

That’s what really happens:

int i = 0x5001;
x["foo"] = std::string();  // in-place creation
x["foo"] = (char)i;        // the string content is replaced with (char)i

Conclusion

We have a nice Rube-Goldberg machine here:

• int is silently converted into char;
• std::map creates an empty string for a key that does not exist;
• std::string::operator= is heavily overloaded, so one of its forms takes a char.

The lesson to learn here is that C++ compilers could waste less time and CPU cycles trying to guess the programmer intent and could benefit from Rust-like explicitness and strictness.

HTTPS

It is a good idea to serve your web-site over HTTPS as soon as you have any kind of authorization/server-side logic (and even if it’s a completely static site, it’s still a good practice to provide users with confidence in the page integrity: plain unencrypted HTTP can be messed up by any intermediate host/proxy/cache). Nowadays, HTTPS is very simple to set up using Let’s Encrypt: it’s a one-off setup procedure with automated renewal of certificates later and no beaurocracy whatsoever (compared to free, but manual and rather beaurocratic procedure of getting StartSSL certificates which I used before).

On the other hand, TLS (the base of HTTPS security) is a complicated protocol with a lot of subtleties to get right: e.g. the server should send the full chain of authority certificates (ideally excluding the root certficate authority which is installed in the OS/browser anyway, to optimize performance), the server should use reliable, modern ciphers and key exchange methods without known vulnerabilites. There are also higher-level attacks like downgrading TLS version by a middleman.

SSLLabs HTTPS test provides a good starting point for learning about these intricacies and vulnerabilities and gives helpful advice for improving the HTTPS setup.

Web security

Even a simple web-service with a couple of forms and actions might provide a surprisingly big attack surface: CSRF, clickjacking, XSS, downgrading SSL/TLS to older versions, third-party Javascript risks (e.g. even if you trust the third-party—which is usually the case!—loading script over plain HTTP might be risky).

A good place to start learning the theory is https://wiki.mozilla.org/WebAppSec and https://www.owasp.org/index.php/Cheat_Sheets.

Mozilla HTTP Observatory is a very helpful tool to check how well your HTTP server setup is equipped against common security hazards: does it send security-relevant HTTP headers, does it have HTTPS redirections from HTTP, etc.

RSS/XML/HTML5 validators

If you have RSS feeds, it’s a good idea to verify it for compliance to standards: https://validator.w3.org/feed/.

User experience

If you want your site to have modern look and good performance, particularly on mobile (which are essential for e.g. good ranking in search results), Google has a helpful tool: PageSpeed Insights.

PageSpeed Insights provides advice about page/assets size (minification of HTML/CSS/JS/images), server response time, page rendering delays, caching and compression setup of the web server and other useful things.

Наприклад, хочемо ми відсортувати список певної довжини insertion sort’ом. Гаразд, пишемо

ins_sort : Ord el => Vect n el -> Vect n el

і говоримо Віму \d (define). Він зв’язується із repl Ідріса і дописує внизу:

ins_sort xs = ?ins_sort_rhs

де ?ins_sort_rhs це дірка, функціональність, уже правильно типізована, тільки ще не написана.

Ясно, що після цього нам хочеться розкласти xs на можливі клози pattern matching. Наводимо курсор на xs, говоримо \c (clauses), Вім переписує визначення в

ins_sort [] = ?ins_sort_rhs_1
ins_sort (x :: xs) = ?ins_sort_rhs_2

Заміняти ?ins_sort_rhs_1 на [] нижче нашої гідності (ми ж наче розумні люди, хай цим займаються інструменти), тому знову можна навести курсор на нього і сказати \o (prOOf search) — і ?ins_sort_rhs_1 сам перепишеться в []

Із ?ins_sort_rhs_2 інструмент уже почувається не таким розумним і доводиться писати код вручну:

ins_sort (x :: xs) = 
   let xs_sorted = ins_sort xs
   in ?sorted_insert

— зате тепер можна навести курсор на ?sorted_insert і зробити із нього нову проблему^W лемму^W функцію за допомогою \l (make Lemma). Вім вставляє нову функцію, якій передає як параметри всі локальні змінні:

sorted_insert : Ord el => 
                 (x : el) -> (xs : Vect k el) -> (xs_sorted : Vect k el)
                 -> Vect (S k) el

Ясно, що старі xs нам тепер не потрібні, доводиться підправляти цей код у

sorted_insert : Ord el => (x : el) -> (xs : Vect k el) -> Vect (S k) el

(де xs — це коротше ім’я для xs_sorted). Знову робимо \d, автоматично пишеться

sorted_insert x xs = ?sorted_insert_rhs

Наводимо курсор на xs, тиснемо \c, створюються клози:

sorted_insert x [] = ?sorted_insert_rhs_1
sorted_insert x (y :: xs) = ?sorted_insert_rhs_2

(зверніть увагу на автоматичне перейменування в y :: xs!).

За допомогою \o ?sorted_insert_rhs_1 автоматично переписується в [x]. Тепер можна зайнятись ?sorted_insert_rhs_2, тепер уже вручну:

sorted_insert x (y :: xs) = 
     if x < y then (x :: y :: xs) else y :: sorted_insert x xs

І вуаля, \r (reload) перевіряє типи в модулі, все нормально. Прямо з Віма пробуємо \e (expression/evaluate?), вводимо ins_sort [5,1,4,3,2] і в вікні Віма (яке з’являється по \i, interactive?) бачимо відповідь:

= [1, 2, 3, 4, 5] : Vect 5 Integer

At the moment I am reading “Software Foundations” by B. Pierce and currently I’m making my way through Imp.v. It shows a tiny language of arithmetical expressions (pure and halting!):

(** Syntax *)
Inductive aexp : Type :=
  | ANum : nat -> aexp
  | AId : id -> aexp  
  | APlus : aexp -> aexp -> aexp
  | AMinus : aexp -> aexp -> aexp
  | AMult : aexp -> aexp -> aexp. 
(** Evaluation *)
Fixpoint aeval (st : state) (a : aexp) : nat :=
  match a with
  | ANum n => n
  | AId x => st x
  | APlus a1 a2 => (aeval st a1) + (aeval st a2)
  | AMinus a1 a2 => (aeval st a1) - (aeval st a2)
  | AMult a1 a2 => (aeval st a1) * (aeval st a2)
  end.

and a tiny virtual machine for evaluating expressions in this language:

Inductive id : Type := Id : nat -> id.

Inductive sinstr : Type :=
  | SPush : nat -> sinstr  (* load a constant on the stack *)
  | SLoad : id -> sinstr   (* load a value from the environment on the stack *)
  | SPlus : sinstr    (* take two values from the stack, push their sum back *)
  | SMinus : sinstr
  | SMult : sinstr. 

It is stack-based (it loads constants or values from an environment on stack and leaves its result there):

Fixpoint s_execute (st: state) (stack: list nat) (prog: list sinstr) : list nat 
(**
  I will not post the body of this function to fulfil 
  the request of the book author to avoid posting solutions 
  to exercises in public; the only thing I say about it is
  that my version ignores stack underflow: the stack is 
  not changed by [SPlus]/[SMult]/[SMinus] if there are
  less than two values on it 
**)

The next necessity is a compiler from aexp to list sinst:

Fixpoint s_compile (e : aexp) : list sinstr := (* censored *)

The climax of this exercise is proving s_compile correct, i.e.

Theorem s_compile_correct : forall (st : state) (e : aexp),
   s_execute st [] (s_compile e) = [ aeval st e ].

(I hope that the following text does not count as a ready solution, rather as a hint).

After some fiddling with the assertion, it became obvious that it would be useful to define small-step lemmas:

Lemma s_execute_step :
  forall (st : state) (s : list nat) (i : sinstr) (p : list sinstr),
    s_execute st s (i :: p) = s_execute st (s_execute st s [i]) p.

Lemma s_execute_map :
  forall (st : state) (s : list nat) (p1 p2 : list sinstr),
    s_execute st s (p1 ++ p2) = s_execute st (s_execute st s p1) p2.

Having been armed with this lemmas, I started tackling the theorem. Quite quickly I decided to assert that the language of aexp cannot modify the stack below its expression (s_compile never produces a list sinstr that underflows the stack):

Lemma aexp_never_underflows:
  forall (st : state) (s : list nat) (e : aexp) (v : nat),
   s_execute st [] (s_compile e) = [v] -> s_execute st s (s_compile e) = v :: s.

and tried to prove this lemma. The funny thing was that this lemma seemed to be dependent on s_compile_correct and vice versa! After some time (I am omitting the related emotional rollercoaster) I realized that it’s not going to work and tried to step back to get the big picture. The lucky idea was to combine the lemmas into one (why not?):

Lemma s_compile_correct_s : 
  forall (st : state) (e : aexp) (s : list nat),
    s_execute st s (s_compile e) = (aeval st e :: s).

As soon as I had it written down, I realized that it’s not a lemma, it’s the theorem I am going after! It’s a more general version of s_compile_correct that I’ve been told to prove! To save your time scrolling back, here is s_compile_correct:

Theorem s_compile_correct :
  forall (st : state) (e : aexp),
    s_execute st [] (s_compile e) = [ aeval st e ].
Proof.
  intros. apply s_compile_correct_s.
  (* I hope this solution is trivial enough to publish it *)
Qed.

s_compile_correct_s was not the easiest thing to prove, but it was doable and straightforward, compared to my previous roundabout wandering in the maze of special cases.

Conclusion: Even though I knew the concept that some problems are much easier in a generalized form, this exercise was a real eye-opener: sometimes it takes a lot of time and effort to recognize a subtle need for generalization.

((¬r ⇒ ¬p ∧ ¬q) ∨ s) ⇔ (p ∨ q ⇒ r ∨ s)

Виписувати таблицю істиності на 16 комбінацій мені стало страшно. Я схитрив і побачив, що для s=1 ця формула тавтологічна, а от для s=0 довелось попереписувати:

(¬r ⇒ ¬p ∧ ¬q) ⇔ (p ∨ q ⇒ r)

помітити, що ¬p ∧ ¬q ⇔ ¬(p ∨ q) за де Морганом, отже

(¬r ⇒ ¬(p ∨ q)) ⇔ (p ∨ q ⇒ r)

довелось ще помітити, що, оскільки (¬x ⇒ ¬y) ⇔ (y ⇒ x), то формула при s=0 також вироджується в тавтологію.

А потім згадав, що є такі чудові штуки, як SMT-солвери, і, розчехливши цю гармату, таки вистрілив по горобцях:

$ rlwrap z3 -smt2 -in
;; описуємо наші змінні:
(declare-const p Bool)
(declare-const q Bool)
(declare-const r Bool)
(declare-const s Bool) 

;; ((¬r ⇒ ¬p ∧ ¬q) ∨ s) ⇔ (p ∨ q ⇒ r ∨ s)
(define-fun lhs () Bool
  (or (=> (not r) (and (not p) (not q))) s))
(define-fun rhs () Bool
  (=> (or p q) (or r s)))
(define-fun conjecture () Bool
  (= lhs rhs))

(push) ;; зберігаємо стан без припущень
(assert conjecture)  ;; робимо припущення, що формула conjecture задовольняється
(check-sat) ;sat    ;; так, існують truth assignments, за яких це так

(pop) ;; позбавляємось припущення істинності conjecture
(assert (not conjecture))
(check-sat) ;unsat   ;; не існує truth assignment, за яких формула заперечується 

;; `conjecture` is satisfiable and its negation is not,
;; so it's valid.(exit)

Спробувати це онлайн можна тут: http://rise4fun.com/z3.

$ cat bf/hello2.bf 

++++++++++[>+++++++>++++++++++>+++>+<<<<-]
 >++.>+.+++++++..+++.>++.<<+++++++++++++++.
 >.+++.------.--------.>+.>.

$ runhaskell bfdecomp.hs < bf/hello2.bf 
  add [0] 10
  bz  [0] f1
b1:
  add [1] 7
  add [2] 10
  add [3] 3
  add [4] 1
  sub [0] 1
  bnz [0] b1
f1:
  add [1] 2
  put [1]
  add [2] 1
  put [2]
  add [2] 7
  put [2]
  put [2]
  add [2] 3
  put [2]
  add [3] 2
  put [3]
  add [1] 15
  put [1]
  put [2]
  add [2] 3
  put [2]
  sub [2] 6
  put [2]
  sub [2] 8
  put [2]
  add [3] 1
  put [3]
  put [4]

I thought that I could recover some patterns pretty easily, for example, [-] is equvalent to setting the current cell to 0. But why it does it? How to determine whether the cycle terminates? What tools are helpful there?

First of all, I don’t want to make Brainfuck programs completely static, my goal here is to recover as many facts about the program as possible and to do maximum partial evaluation. In general, if a progam does not depend on input, it’s even safe to “replay” it and to emulate its behavior with put $val1; put $val2; .... But here is the first hurdle: what if it is an endless loop? First we have to prove that the loop terminates. That part in general is undecidable (it is basically the halting problem), but some partial evaluations are quite possible.

Recently I had a look at Facebook Infer, a fast static analyser for Java/C programs that infers pre/postconditions of functions on its own. It’s a pretty neat idea and the separation logic behind it looks attractive, so I wanted to apply some of it to the primitiveness of BrainFck.

Let’s apply Hoare triples here. First of all, we have to infer effects of primitive operations (let p be the data pointer, [p] - the contents of the current cell) like that:

add [p] n \(p_{post} = p_{pre}, [p]_{post} = [p]_{pre} + n;\)

Then we need our program to understand arithmetics: for example, it must know that add [p] n, applied to [p] m times, is equal to add [p] n*m. Then we need to be able to reason about variables: e.g. >++++<- does not move the data pointer, or, in other words, we must be able to figure out that \(a_{pre} = a_{0}\), \(a_1 = a_0 + 1\), \(a_2 = a_1 - 1 = a_0 + 1 - 1 = a_0\), so \(a_{pre}=a_{post}\); the contents of cells must obey rules: \([a_0]_{post} = [a_0]_{pre} - 1\), \([a_1]_{post} = [a_1]_{pre} + 4\)

get/put are not like other operations: we have to preserve their effects. Any value introduced by get must be “poisoned”, we only know that it may be in range from 0 to 255 and is determined at run-time. When multiplied with another value, it must “poison” the result.

When I had a look at BF self-interpreter, I was a little shocked: I thought that people are not that smart to smash runtime pointers so freely with constructs like [<] (“move left until a cell with zero in it found”) and [-[->]]. Now I have to think how to handle such situations without going crazy and how to recover facts about non-zero data regions. But it looks like here I will fight the concrete wall of Turing completeness