developersguide:liveupdate
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision | Next revision Both sides next revision | ||
|
developersguide:liveupdate [2016/01/22 12:26] dcvmoole new instructions for minix-trunk as of git-8deb69f |
developersguide:liveupdate [2018/09/03 13:25] stux fix formatting in the bottom half of the article! |
||
|---|---|---|---|
| Line 424: | Line 424: | ||
| The following piece of pseudocode represents a simplified and flattened version of the general structure of each system service: | The following piece of pseudocode represents a simplified and flattened version of the general structure of each system service: | ||
| - | <code> | + | ''<nowiki> |
| main: | main: | ||
| # initialization | # initialization | ||
| Line 446: | Line 446: | ||
| continue | continue | ||
| handle message | handle message | ||
| - | </code> | + | </nowiki>'' |
| As can be seen, the service's initialization code starts by learning from RS what type of initialization it should perform. | As can be seen, the service's initialization code starts by learning from RS what type of initialization it should perform. | ||
| Line 627: | Line 627: | ||
| * **[ERROR]** uncaught ptr with violations. Current state element: | * **[ERROR]** uncaught ptr with violations. Current state element: | ||
| - | * SELEMENT: (parent=sbuf.1900354961, num=1, depth=0, address=0xdfb760a8, **name**=**sbuf**.1900354961, type=TYPE: (id=53 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=01010000, values=%%''%%, type_str=i8/**char%%*%%**)) | + | * SELEMENT: ''<nowiki>(parent=sbuf.1900354961, num=1, depth=0, address=0xdfb760a8, **name**=**sbuf**.1900354961, type=TYPE: (id=53 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=01010000, values=%%''%%, type_str=i8/**char%%*%%**))</nowiki>'' |
| - | * SEL_ANALYZED: (num=1, type=ptr, flags(DIVW)=1110, **value**=**0x080cb49f**, trg_name=, trg_offset=0, trg_flags(RL)=D0, trg_selements=(#1|0: 1|p=SELEMENT: (parent=???, num=0, depth=0, address=0x00000000, name=???, type=TYPE: (id=0 , name=**UNKNOWN_TYPE**, size=0, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=10000000, values=%%''%%, type_str=UNKNOWN_TYPE/UNKNOWN_TYPE)))) | + | * SEL_ANALYZED: ''<nowiki>(num=1, type=ptr, flags(DIVW)=1110, **value**=**0x080cb49f**, trg_name=, trg_offset=0, trg_flags(RL)=D0, trg_selements=(#1|0: 1|p=SELEMENT: (parent=???, num=0, depth=0, address=0x00000000, name=???, type=TYPE: (id=0 , name=**UNKNOWN_TYPE**, size=0, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=10000000, values=%%''%%, type_str=UNKNOWN_TYPE/UNKNOWN_TYPE))))</nowiki>'' |
| - | * SEL_STATS: (type=ptr, trg_flags(RL)=D0, ptr_found=1, **unknown_found=1**, violations=1) | + | * SEL_STATS: ''<nowiki>(type=ptr, trg_flags(RL)=D0, ptr_found=1, **unknown_found=1**, violations=1)</nowiki>'' |
| In this case, the global variable **sbuf** (suffixed with a tag to make its name unique) is a char* pointer to location 0x080cb49f. Since the magic runtime library knows no type information about this target (//trg//) memory location, it marks the location with the placeholder type UNKNOWN_TYPE and aborts state transfer because an unknown type was found. Another example: | In this case, the global variable **sbuf** (suffixed with a tag to make its name unique) is a char* pointer to location 0x080cb49f. Since the magic runtime library knows no type information about this target (//trg//) memory location, it marks the location with the placeholder type UNKNOWN_TYPE and aborts state transfer because an unknown type was found. Another example: | ||
| * **[ERROR]** uncaught ptr with violations. Current state element: | * **[ERROR]** uncaught ptr with violations. Current state element: | ||
| - | * SELEMENT: (parent=inode.3951291702, num=80, depth=2, address=0xdfbe3210, **name**=**inode.3951291702/4/i_data**, type=TYPE: (id=61 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=01010000, values=%%''%%, type_str=i8/**char%%*%%**)) | + | * SELEMENT: ''<nowiki>(parent=inode.3951291702, num=80, depth=2, address=0xdfbe3210, **name**=**inode.3951291702/4/i_data**, type=TYPE: (id=61 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=01010000, values=%%''%%, type_str=i8/**char%%*%%**))</nowiki>'' |
| - | * SEL_ANALYZED: (num=80, type=ptr, flags(DIVW)=1110, **value**=**0x08108098**, trg_name=, trg_offset=0, trg_flags(RL)=H0, trg_selements=(#1|0: 1|p=SELEMENT: (parent=???, num=0, depth=0, address=0x00000000, name=???, type=TYPE: (id=0 , name=**UNKNOWN_TYPE**, size=0, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=10000000, values=%%''%%, type_str=UNKNOWN_TYPE/UNKNOWN_TYPE)))) | + | * SEL_ANALYZED: ''<nowiki>(num=80, type=ptr, flags(DIVW)=1110, **value**=**0x08108098**, trg_name=, trg_offset=0, trg_flags(RL)=H0, trg_selements=(#1|0: 1|p=SELEMENT: (parent=???, num=0, depth=0, address=0x00000000, name=???, type=TYPE: (id=0 , name=**UNKNOWN_TYPE**, size=0, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=10000000, values=%%''%%, type_str=UNKNOWN_TYPE/UNKNOWN_TYPE))))</nowiki>'' |
| - | * SEL_STATS: (type=ptr, trg_flags(RL)=H0, ptr_found=1, **unknown_found=1**, violations=1) | + | * SEL_STATS: ''<nowiki>(type=ptr, trg_flags(RL)=H0, ptr_found=1, **unknown_found=1**, violations=1)</nowiki>'' |
| In this case, the **i_data** field of the fifth element (**/4/**) of the global **inode** structure, also a char* pointer, is pointing to address 0x08108098 which is unknown to libmagicrt. The pointer address typically allows one to determine what kind of memory it is, by means of the memory sections of the process. In this particular example, the address was somewhat higher than the service's data end, thus suggesting the memory pointed to is heap memory. This matched with the source code of the service (PFS, the Pipe File Server), which dynamically allocates and frees the i_data buffers using malloc(3) and free(3). | In this case, the **i_data** field of the fifth element (**/4/**) of the global **inode** structure, also a char* pointer, is pointing to address 0x08108098 which is unknown to libmagicrt. The pointer address typically allows one to determine what kind of memory it is, by means of the memory sections of the process. In this particular example, the address was somewhat higher than the service's data end, thus suggesting the memory pointed to is heap memory. This matched with the source code of the service (PFS, the Pipe File Server), which dynamically allocates and frees the i_data buffers using malloc(3) and free(3). | ||
| Line 665: | Line 665: | ||
| * **[ERROR]** uncaught ptr with violations. Current state element: | * **[ERROR]** uncaught ptr with violations. Current state element: | ||
| - | * SELEMENT: (parent=ds_subs.1944246923, num=9, depth=3, address=0xdfbe6108, name=**ds_subs.1944246923/0/regex/re_g**, type=TYPE: (id=18 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=00000000, values=%%''%%, type_str=opaque*)) | + | * SELEMENT: ''<nowiki>(parent=ds_subs.1944246923, num=9, depth=3, address=0xdfbe6108, name=**ds_subs.1944246923/0/regex/re_g**, type=TYPE: (id=18 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=00000000, values=%%''%%, type_str=opaque*))</nowiki>'' |
| - | * SEL_ANALYZED: (num=9, type=ptr, flags(DIVW)=1110, **value**=**0x08111000**, trg_name=, trg_offset=0, trg_flags(RL)=, trg_selements=(#1|0: 1|p=SELEMENT: (parent=???, num=0, depth=0, address=0x00000000, name=???, type=TYPE: (id=0 , name=**UNKNOWN_TYPE**, size=0, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=10000000, values=%%''%%, type_str=UNKNOWN_TYPE/UNKNOWN_TYPE)))) | + | * SEL_ANALYZED: ''<nowiki>(num=9, type=ptr, flags(DIVW)=1110, **value**=**0x08111000**, trg_name=, trg_offset=0, trg_flags(RL)=, trg_selements=(#1|0: 1|p=SELEMENT: (parent=???, num=0, depth=0, address=0x00000000, name=???, type=TYPE: (id=0 , name=**UNKNOWN_TYPE**, size=0, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=10000000, values=%%''%%, type_str=UNKNOWN_TYPE/UNKNOWN_TYPE))))</nowiki>'' |
| - | * SEL_STATS: (type=ptr, ptr_found=1, **unknown_found=1**, violations=1) | + | * SEL_STATS: ''<nowiki>(type=ptr, ptr_found=1, **unknown_found=1**, violations=1)</nowiki>'' |
| In this case, the pointer **ds_subs[0].regex.re_g** ended up pointing to the unknown heap-section value of 0x08111000. We worked around this issue by forcing DS to use the targets of the weak aliases, _regcomp and _regfree, rather than their original names, using Makefile hacks. | In this case, the pointer **ds_subs[0].regex.re_g** ended up pointing to the unknown heap-section value of 0x08111000. We worked around this issue by forcing DS to use the targets of the weak aliases, _regcomp and _regfree, rather than their original names, using Makefile hacks. | ||
| Line 676: | Line 676: | ||
| * **[ERROR]** uncaught ptr with violations. Current state element: | * **[ERROR]** uncaught ptr with violations. Current state element: | ||
| - | * SELEMENT: (parent=_ctype_tab_, num=1, depth=0, address=0xdfb760a8, **name**=**_ctype_tab_**, type=TYPE: (id=204 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=11000000, values=%%''%%, type_str=i16/unsigned short*)) | + | * SELEMENT: ''<nowiki>(parent=_ctype_tab_, num=1, depth=0, address=0xdfb760a8, **name**=**_ctype_tab_**, type=TYPE: (id=204 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=11000000, values=%%''%%, type_str=i16/unsigned short*))</nowiki>'' |
| - | * SEL_ANALYZED: (num=1, type=ptr, flags(DIVW)=1110, **value**=**0x0809ccb6**, trg_name=, trg_offset=0, trg_flags(RL)=, trg_selements=(#1|0: 1|p=SELEMENT: (parent=???, num=0, depth=0, address=0x00000000, name=???, type=TYPE: (id=0 , name=**UNKNOWN_TYPE**, size=0, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=10000000, values=%%''%%, type_str=UNKNOWN_TYPE/UNKNOWN_TYPE)))) | + | * SEL_ANALYZED: ''<nowiki>(num=1, type=ptr, flags(DIVW)=1110, **value**=**0x0809ccb6**, trg_name=, trg_offset=0, trg_flags(RL)=, trg_selements=(#1|0: 1|p=SELEMENT: (parent=???, num=0, depth=0, address=0x00000000, name=???, type=TYPE: (id=0 , name=**UNKNOWN_TYPE**, size=0, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=10000000, values=%%''%%, type_str=UNKNOWN_TYPE/UNKNOWN_TYPE))))</nowiki>'' |
| - | * SEL_STATS: (type=ptr, ptr_found=1, **unknown_found=1**, violations=1) | + | * SEL_STATS: ''<nowiki>(type=ptr, ptr_found=1, **unknown_found=1**, violations=1)</nowiki>'' |
| In this particular failure case, the global **_ctype_tab_** variable pointed into another global variable, at location 0x0809ccb6 the data section. The other global variable was invisible to the magic pass, so no **sentry** object could be created for it. As a result, libmagicrt did not know about the target of the pointer. The _ctype_tab_ variable itself was used by the ''<ctype.h>'' isalpha(3) (etc) set of macros from within libmagicrt. We worked around this issue by putting our own replacement set of macros in libmagicrt instead. | In this particular failure case, the global **_ctype_tab_** variable pointed into another global variable, at location 0x0809ccb6 the data section. The other global variable was invisible to the magic pass, so no **sentry** object could be created for it. As a result, libmagicrt did not know about the target of the pointer. The _ctype_tab_ variable itself was used by the ''<ctype.h>'' isalpha(3) (etc) set of macros from within libmagicrt. We worked around this issue by putting our own replacement set of macros in libmagicrt instead. | ||
| Line 693: | Line 693: | ||
| * **[ERROR]** uncaught ptr with violations. Current state element: | * **[ERROR]** uncaught ptr with violations. Current state element: | ||
| - | * SELEMENT: (parent=timers.515278380, num=1, depth=0, address=0xdfb760a8, **name**=**timers**.515278380, type=TYPE: (id=96 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=01000000, values=%%''%%, **type_str**={ $minix_timer tmr_next \2, tmr_exp_time i32/long unsigned int, **tmr_func opaque%%*%%**, tmr_arg { (U) $ixfer_tmr_arg_t ta_int i32/int } }*)) | + | * SELEMENT: ''<nowiki>(parent=timers.515278380, num=1, depth=0, address=0xdfb760a8, **name**=**timers**.515278380, type=TYPE: (id=96 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=01000000, values=%%''%%, **type_str**={ $minix_timer tmr_next \2, tmr_exp_time i32/long unsigned int, **tmr_func opaque%%*%%**, tmr_arg { (U) $ixfer_tmr_arg_t ta_int i32/int } }*))</nowiki>'' |
| - | * SEL_ANALYZED: (num=1, type=ptr, flags(DIVW)=1110, value=0x08147460, trg_name=mproc, trg_offset=274616, trg_flags(RL)=D0, trg_selements=(**#2**|0: **1**|o=SELEMENT: (parent=mproc, num=0, depth=0, address=0x08147460, name=**mproc/143/mp_timer**, type=TYPE: (id=38 , name=minix_timer, size=16, num_child_types=4, type_id=9, bit_width=0, flags(ERDIVvUP)=00000000, values=%%''%%, names='minix_timer_t|minix_timer', **type_str**={ $minix_timer tmr_next { $minix_timer tmr_next \2, tmr_exp_time i32/long unsigned int, tmr_func hash_3792421438/*, tmr_arg { (U) $ixfer_tmr_arg_t ta_int i32/int } }*, tmr_exp_time i32/long unsigned int, **tmr_func hash_3792421438/%%*%%**, tmr_arg { (U) $ixfer_tmr_arg_t ta_int i32/int } })), **2**|o=SELEMENT: (parent=mproc, num=0, depth=0, address=0x08147460, name=mproc/143/mp_timer/tmr_next, type=TYPE: (id=37 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=00000000, values=%%''%%, **type_str**={ $minix_timer tmr_next \2, tmr_exp_time i32/long unsigned int, **tmr_func hash_3792421438/%%*%%**, tmr_arg { (U) $ixfer_tmr_arg_t ta_int i32/int } }*)))) | + | * SEL_ANALYZED: ''<nowiki>(num=1, type=ptr, flags(DIVW)=1110, value=0x08147460, trg_name=mproc, trg_offset=274616, trg_flags(RL)=D0, trg_selements=(**#2**|0: **1**|o=SELEMENT: (parent=mproc, num=0, depth=0, address=0x08147460, name=**mproc/143/mp_timer**, type=TYPE: (id=38 , name=minix_timer, size=16, num_child_types=4, type_id=9, bit_width=0, flags(ERDIVvUP)=00000000, values=%%''%%, names='minix_timer_t|minix_timer', **type_str**={ $minix_timer tmr_next { $minix_timer tmr_next \2, tmr_exp_time i32/long unsigned int, tmr_func hash_3792421438/*, tmr_arg { (U) $ixfer_tmr_arg_t ta_int i32/int } }*, tmr_exp_time i32/long unsigned int, **tmr_func hash_3792421438/%%*%%**, tmr_arg { (U) $ixfer_tmr_arg_t ta_int i32/int } })), **2**|o=SELEMENT: (parent=mproc, num=0, depth=0, address=0x08147460, name=mproc/143/mp_timer/tmr_next, type=TYPE: (id=37 , name=, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=00000000, values=%%''%%, **type_str**={ $minix_timer tmr_next \2, tmr_exp_time i32/long unsigned int, **tmr_func hash_3792421438/%%*%%**, tmr_arg { (U) $ixfer_tmr_arg_t ta_int i32/int } }*))))</nowiki>'' |
| - | * SEL_STATS: (type=ptr, trg_flags(RL)=D0, ptr_found=1, **other_types_found=1**, violations=1) | + | * SEL_STATS: ''<nowiki>(type=ptr, trg_flags(RL)=D0, ptr_found=1, **other_types_found=1**, violations=1)</nowiki>'' |
| In this case, the analysis failed on the global **timers** variable. The analysis dump shows that two matching types (**#2**) were found, both associated with the **mproc[143].mp_timer** structure field, but neither type matched the type of the pointer. A closer look at the textual representations of the pointer type (the **type_str** of the primary //selement//) and of the data types (the //type_str// of the target //selement//s) reveals that there is only one difference between the two: the **tmr_func** field of the structure type to which the //timers// variable should point is an **opaque** pointer, whereas the same //tmr_func// field of the target structures is a particular function pointer (to a function referred to as **hash_3792421438**). The remainder of the types are the same. | In this case, the analysis failed on the global **timers** variable. The analysis dump shows that two matching types (**#2**) were found, both associated with the **mproc[143].mp_timer** structure field, but neither type matched the type of the pointer. A closer look at the textual representations of the pointer type (the **type_str** of the primary //selement//) and of the data types (the //type_str// of the target //selement//s) reveals that there is only one difference between the two: the **tmr_func** field of the structure type to which the //timers// variable should point is an **opaque** pointer, whereas the same //tmr_func// field of the target structures is a particular function pointer (to a function referred to as **hash_3792421438**). The remainder of the types are the same. | ||
| Line 706: | Line 706: | ||
| * **[ERROR]** uncaught ptr with violations. Current state element: | * **[ERROR]** uncaught ptr with violations. Current state element: | ||
| - | * SELEMENT: (parent=sched_timer.29458437, num=4, depth=1, address=0xdfbe70b0, **name**=**sched_timer.29458437/tmr_func**, type=TYPE: (id=17 , name=tmr_func_t, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=00000000, values=%%''%%, type_str=**opaque%%*%%**)) | + | * SELEMENT: ''<nowiki>(parent=sched_timer.29458437, num=4, depth=1, address=0xdfbe70b0, **name**=**sched_timer.29458437/tmr_func**, type=TYPE: (id=17 , name=tmr_func_t, size=4, num_child_types=1, type_id=10, bit_width=0, flags(ERDIVvUP)=00000000, values=%%''%%, type_str=**opaque%%*%%**))</nowiki>'' |
| - | * SEL_ANALYZED: (num=4, type=ptr, flags(DIVW)=1110, value=0x08048dc0, trg_name=**balance_queues**.29458437, trg_offset=0, trg_flags(RL)=T0, trg_selements=(#1|0: 1|o=SELEMENT: (parent=???, num=0, depth=0, address=0x08048dc0, name=???, type=TYPE: (id=119 , name=, size=1, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=11000001, values=%%''%%, type_str=**hash_3792445575/**)))) | + | * SEL_ANALYZED: ''<nowiki>(num=4, type=ptr, flags(DIVW)=1110, value=0x08048dc0, trg_name=**balance_queues**.29458437, trg_offset=0, trg_flags(RL)=T0, trg_selements=(#1|0: 1|o=SELEMENT: (parent=???, num=0, depth=0, address=0x08048dc0, name=???, type=TYPE: (id=119 , name=, size=1, num_child_types=0, type_id=4, bit_width=0, flags(ERDIVvUP)=11000001, values=%%''%%, type_str=**hash_3792445575/**))))</nowiki>'' |
| - | * SEL_STATS: (type=ptr, trg_flags(RL)=T0, ptr_found=1, **other_types_found=1**, violations=1) | + | * SEL_STATS: ''<nowiki>(type=ptr, trg_flags(RL)=T0, ptr_found=1, **other_types_found=1**, violations=1)</nowiki>'' |
| In this case, the type mismatch was not between two structures that differed in opaque fields, but between two function pointers themselves: the function pointer in **sched_timer.tmr_func**, and the function it is pointing to, **balance_queues**. Registering these types as compatible would result in much more complexity in the magic pass, and likely still not resolve the more general problem of opaque pointers. This is currently one of the open issues, and we believe that another approach would be more viable; see below. In this particular case, it turned out that the sched service did not need to use timers at all, and we simplified it by getting rid of its use of timers altogether. Obviously, adapting the actual functionality of a service to allow for state transfer is not always an option, nor is it generally the right approach: the core code of system services should not have to be (re)written specifically to allow for state transfer. | In this case, the type mismatch was not between two structures that differed in opaque fields, but between two function pointers themselves: the function pointer in **sched_timer.tmr_func**, and the function it is pointing to, **balance_queues**. Registering these types as compatible would result in much more complexity in the magic pass, and likely still not resolve the more general problem of opaque pointers. This is currently one of the open issues, and we believe that another approach would be more viable; see below. In this particular case, it turned out that the sched service did not need to use timers at all, and we simplified it by getting rid of its use of timers altogether. Obviously, adapting the actual functionality of a service to allow for state transfer is not always an option, nor is it generally the right approach: the core code of system services should not have to be (re)written specifically to allow for state transfer. | ||
| Line 865: | Line 865: | ||
| * Cristiano Giuffrida, [[http://www.minix3.org/theses/Cristiano_Giuffrida_PhD_thesis.pdf|Safe and Automatic Live Update]], Ph.D. thesis, 2014 | * Cristiano Giuffrida, [[http://www.minix3.org/theses/Cristiano_Giuffrida_PhD_thesis.pdf|Safe and Automatic Live Update]], Ph.D. thesis, 2014 | ||
| + | |||
developersguide/liveupdate.txt · Last modified: 2022/02/12 22:42 by stux
