README.md 3.15 KB
Newer Older
Dave Griffiths's avatar
docs    
Dave Griffiths committed
1
2
3
Symbai/Starwisp
===============

Dave Griffiths's avatar
docs    
Dave Griffiths committed
4
This is a data syncing application using a single raspberry pi for
Dave Griffiths's avatar
docs    
Dave Griffiths committed
5
syncronisation between many android tablets. Designed for recording
Dave Griffiths's avatar
docs    
Dave Griffiths committed
6
7
8
9
behavioural observations in field sites in remote locations.

Designed to work off grid and off line, for example working from solar
powered devices and using an ad hoc wifi connection.
Dave Griffiths's avatar
docs    
Dave Griffiths committed
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

Development notes
=================

Data model
----------

The sqlite database is using an entity-attribute-value system so data types can
be added dynamically, with only an android application update required.

Each item in the database is an entity, which contains a version number,
a local id (the primary key) which is defined on the local database only
and a unique-id which is common to all databases across the
network. Entities are split into different types, and are associated
with attributes which make up their contents. The attributes of the
entity are stored in key-type-value attributes (ktv). Entities are
comprised of multiple ktv attributes.

When an entity is edited, it's flagged as dirty - along with the
individual attributes which are changed.

The entity dirty flag is used to determine which entities need to be
sent to the server when syncing, and the attribute's dirty flags
indicate which individual items need sending.

The server increments the entity version number on reciept of a changed
values, which is used to determine which entities are needed updating on
other clients.

In this way, data changes are merged and propagated through the system.

Dave Griffiths's avatar
docs    
Dave Griffiths committed
41
Files can be associated with attributes, and are synced (e.g. images)
Dave Griffiths's avatar
docs    
Dave Griffiths committed
42

Dave Griffiths's avatar
docs    
Dave Griffiths committed
43
44
Syncing algorithm
-----------------
Dave Griffiths's avatar
docs    
Dave Griffiths committed
45

Dave Griffiths's avatar
docs    
Dave Griffiths committed
46
Sending data from client to server is all about dirty flags:
Dave Griffiths's avatar
docs    
Dave Griffiths committed
47

Dave Griffiths's avatar
docs    
Dave Griffiths committed
48
| Android (client)              |   Raspberry Pi (server)      |
Dave Griffiths's avatar
docs    
Dave Griffiths committed
49
50
51
|-------------------------------|------------------------------|
| Data edited or created - cause entities and attributes to be flagged as 'dirty' |      |
| For each dirty entity, send all dirty attributes | Return message to confirm reciept |
Dave Griffiths's avatar
docs    
Dave Griffiths committed
52
| Clean dirty flags on returned message | |
Dave Griffiths's avatar
docs    
Dave Griffiths committed
53
54
|                               | Overwrite all dirty attributes, increment version number |

Dave Griffiths's avatar
docs    
Dave Griffiths committed
55
56
Receiving changes from the server to the client is all about version numbers:

Dave Griffiths's avatar
docs    
Dave Griffiths committed
57
| Android (client)              |   Raspberry Pi (server)      |
Dave Griffiths's avatar
docs    
Dave Griffiths committed
58
59
60
61
62
63
|-------------------------------|------------------------------|
| Request all entity version numbers | Send versions...      |
| Check each local entity, request updates | Send complete entity, with all attributes |
| Overwrite/add new entities    |  |


Dave Griffiths's avatar
docs    
Dave Griffiths committed
64
65
Code dependancies
-----------------
Dave Griffiths's avatar
docs    
Dave Griffiths committed
66

Dave Griffiths's avatar
docs    
Dave Griffiths committed
67
Common code is stored in /eavdb and is platform independant, running on android (tinyscheme) and raspberry pi (racket).
Dave Griffiths's avatar
docs    
Dave Griffiths committed
68

Dave Griffiths's avatar
docs    
Dave Griffiths committed
69
70
71
72
         ktv ---> ktv-list      sql/racket-fix
          \         |              /
           \        V             /
            --> entity-values  <--
Dave Griffiths's avatar
docs    
Dave Griffiths committed
73
74
75
76
77
78
79
80
81
82
83
84
                 |        |
                 V        V
        entity-insert    entity-get
                 |        |
                 V        V
                entity-update
                 |   |    |
                 V   |    V
        entity-sync  |   entity-filter
                 |   |    |
                 V   V    V
                    eavdb