xATC Proposal

xATC is a proposed system to supersede conventional ATC, by incorporating and taking advantage of new features, like:

• Approach callbacks
• Train variables(line, routingcode, texts)
• Timetables (planned) and Railway Time scheduling
• programmatically splitting and joining trains

This is a Very WIP, experimental draft which is nowhere near final. Suggestions, ideas and improvements are welcome. If you have a suggestion, just add it on this page and add your signature (second-last button in the editor toolbar above).

At the moment, there exist two systems for controlling trains automatedly. They somewhat work hand-in-hand, but have individual design flaws:

This is the first ATC system that ever existed in advtrains, and has not changed much since then.

• Commands are purely assigned to the train. The train is responsible for parsing and executing the commands.
• This means that there's no way to determine where the command came from
• The syntax relies on 1-letter command codes and simple RegEx matching. It is not extensible.
• In particular, it is impossible to integrate more advanced commands like setting in/outside text or setting routes
• Commands have to be one-liners

This second automation system is powerful, but not very thought through. It uses plain Lua scripts in a slightly sandboxed environment.

• Commands are purely assigned to rails, and can not affect trains that are not standing on the rail (except for approach callbacks which use an ugly workaround)
• Insufficient sandboxing (Multiple DoS attacks possible)

• MUST support everything conventional ATC can do
• SHOULD support everything LuaATC can do
• Should be easy to use for newcomers (not pure Lua)
• Setups should be copyable, and there should be a way to modularize setups (“templates”/“functions”)
• Commands are given out by stationary components, but are associated and functional even when the train is no longer at the commanding location (like LuaATC currently enforces). In a later stage, it should also be possible to assign xATC programs purely to trains (see H#124)

The following things should be doable through xATC:

• Controlling current train target speed, open/close doors, reverse train
• Using the LZB mechanism to brake a train in advance/make it stop
• Schedule execution of further commands by both simple time delay (like “D”) and Railway Time
• Setting train parameters (line, rc, inside/outside text, custom values)
• Allow for splitting/joining trains and managing train sections (different parts of the train designated for different destinations, planned)
• Control interlocking (set/cancel routes, enable/disable ARS on train)
• Interface to external components (e.g. digilines)
• Be extensible to new features introduced over time

Half a year of rethinking xATC has brought some new approaches:

xATC can draw inspiration from other “programming languages”:

• Scratch (https://scratch.mit.edu/): Event blocks work quite similar to individual diagrams in Scratch: they start with an event and execute concurrently when the event fires; alternatively the code can also wait for other events during execution
• Python asyncio: the “await” keyword, used to “wait” for a Future to complete, and the concept of Futures in general

xATC code is organized in “Event Blocks”.

An Event Block consists of an event identifier and a set of code lines. There are built-in identifiers that depend on the context, and the user can define and call own event identifiers.

Each xATC snippet is run in a context.

A new context is instantiated when a context does not exist for a particular combination of involved entities. For example, when a train approaches an xATC rail, a new context for this combination of train and xATC rail is created, unless one already exists.

A context is destroyed when all event blocks in the context are completed and no future events are scheduled for it.

For the first implementation, only xATC rail contexts will exist, however this should be extensible to support train-only contexts, timetable system contexts etc. in the future. Depending on this context type, special variables are bound (for xatc rail contexts, the variables “train” and “rail”).

A context is not created at all when an event is fired on a not-yet-existing context, but the associated xATC snippet has no handling event block for this event.

The following high-level operations exist (analogous to Scratch “blocks”):

• Property/Variable assignment (entity.property = expression)
  • Used for example for train inside/outside texts
• Local variable creation/assignment (local var = expression)
  • Local vars are only valid in this event block
• Wait for a Future to complete (await expression_returning_a_future)
  • this can be waiting for a certain time, waiting for a signal to turn green etc
• Wait for an event to occur in the current context (await event(“event_identifier”))
• Fire an event in the current context (fire(“event_identifier”))
• Fire an event in a foreign context (foreign_context.fire(“event_identifier”))
• Call another built-in/user-defined function (function_name(arguments))
• Control structures: If/Else, for loop?
• Exit this event block (return)
• Destroy the context, killing this and all other event blocks in it (destroy())

This concept does no longer allow for directly “scheduling” events later in a context, but I currently see little need for this as long as you can “wait until time” using await. If it becomes necessary, we can just add a schedule(time, event_identifier) function.

fire() makes it possible to branch execution in multiple concurrent threads.

Event blocks are executed concurrent, but not multithreaded. Since event blocks will not perform intensive calculations but instead yield very often (via await), every time event blocks (within a context) become ready to continue execution, they are executed consecutively until the next yield. If we ever support loops, they would also yield implicitly every iteration (but instantly become ready again).

The code written in xATC is untrusted, since it is possibly written by untrusted users. We therefore build a somewhat custom Lua interpreter. A plan for this looks as follows:

• Every line is considered an instruction or a block start/end (we won't support ; to separate multiple commands per line for now)
• The global instruction structure is parsed by custom code
• Individual instructions are left as-is and are compiled to Lua directly.
• A special xATC-to-Lua compiler will find out where yield points are, split the xatc instructions into consecutive instructions without yield, and compile these somehow as individual Lua functions. This must be done in the way that the yield point the code currently is at can be saved persistently. We assume that execution does not halt between yield points.
• Instructions are only allowed to be in “Assignment” or “Function Call” format. It should not be possible to construct things like '(function() evil_stuff end)()'

As for the xATC rail, it will emit the following events (in typical order of occurence)

• approach: when train is approaching
• enter: when train is entering the rail
• leave: when train is leaving the rail

Special variables bound are “train” and “rail”. Aside from their built-in properties, each of them has a “data” property that may be used to hold custom data.

This could be a simple stop rail:

on approach:
  if train.line == "3" then -- "line" is a property of the train
    set_point_speed_limit(2) -- this is a context function since it depends on both train and rail
    train.disable_ars = true
  end
-- this code does not yield, it directly terminates
-- At this point, no event block is active anymore and the context is destroyed

on enter:
  if train.line ~= "3" then
    return  -- Do not do anything if this is not line 3
  end
  await train.brake(0) -- instruct the train to brake to speed 0, and wait for it
  await train.open_doors_right() -- open the doors and wait for it
  await delay(20) -- wait at least 20 seconds before continuing
  train.disable_ars = false
  await train.next_signal.wait_for_proceed() -- wait until the signal ahead shows a non-danger aspect
  await train.close_doors()
  train.speed(15) -- depart from the station

More sophisticated: a station where two trains wait for each other to arrive, then wait 20sec each and depart:

on approach:
  set_point_speed_limit(2) -- this is a context function since it depends on both train and rail
  train.disable_ars = true

on enter:
  await train.brake(0)
  await train.open_doors_right()
  
  local other_track = get_xatc_rail("OtherTrack")
  
  rail.data["train_arrived"] = true
  -- if nothing yields (no await and no loops) we have no preemption, so race conditions can not occur.
  if other_track.data["train_present"] then
    -- the train is already present on the other track and
    -- is waiting to receive our arrival event
    other_track.fire_all("other_train_arrived") -- fires the event in all active contexts of that rail
  else
    -- the other train has not arrived yet.
    -- wait for the "other_train_arrived" event that it will send us when it arrives
    await event("other_train_arrived")
    -- note that we don't have a dedicated event block for that event, we just wait for it in here
  end
  -- when we get here, both trains have arrived.
  rail.data["train_arrived"] = false
  
  await delay(20) -- wait at least 20 seconds before continuing
  train.disable_ars = false
  await train.next_signal.wait_for_proceed() -- wait until the signal ahead shows a non-danger aspect
  await train.close_doors()
  train.speed(15) -- depart from the station

An alternative way to write the last example:

on approach:
  set_point_speed_limit(2) -- this is a context function since it depends on both train and rail
  train.disable_ars = true

on enter:
  await train.brake(0)
  await train.open_doors_right()
  
  rail.data["train_arrived"] = true
  
  local other_track = get_xatc_rail("OtherTrack")
  if other_track.data["train_present"] then
    -- the train is already present on the other track and
    -- is waiting to receive our arrival event
    other_track.fire_all("other_train_arrived") -- fires the event in all active contexts of that rail
    fire("other_train_arrived") -- and fire the event locally for us
  end

-- this event gets always fired by the train arriving second
on other_train_arrived:
  rail.data["train_arrived"] = false
  
  await delay(20) -- wait at least 20 seconds before continuing
  train.disable_ars = false
  await train.next_signal.wait_for_proceed() -- wait until the signal ahead shows a non-danger aspect
  await train.close_doors()
  train.speed(15) -- depart from the station

Note that this whole example can probably benefit from a semaphore concept. This is TBD.

xATC will use a subset of Lua to express actions. The following things are restricted:

• “while” and “repeat/until” loops are not supported
• “for” loops only allow using a counter variable, not an iterator function
• The global environment is not writable. A number of standard variables and functions exists to be used.

xATC code is organized in so-called “events”. An event can be a builtin event or a user-defined event. The RWT scheduler, which has a builtin DOS protection, will serve as event queue.

An xATC code snippet would, for example, look like this:

On Approach:
train.text_outside = "E3 - Trisiston\nvia Tanh Cliffs"
train.text_inside = "Next Stop: Euler Street\nTerminus, please get off."
train.ars = false
train.stop()   -- makes train stop at the rail and emits "Stop" afterwards
rail.digiline_send("DFI-controller", "Arr:E3W")

On Stop:
train.reverse()
train.text_inside = "Euler Street"
train.doors("R")
schedule("+02;00", "Depart-Ready") -- Schedule departure in 2 minutes
rail.digiline_send("DFI-controller", "Stp:E3W")

On Depart-Ready:
signal.ars().wait("Depart") -- Tell the next signal to do ARS and emit "Depart" when route is set

On Depart:
train.doors("C").wait() -- Wait in-place (without another event) for doors to close
train.text_inside = "Welcome on board! Next stop: Mountain South"
train.ars = true
train.speed()
rail.digiline_send("DFI-controller", "Dep:E3W")

Notable things:

• Events are denoted by “On Eventname:”. They are not lua-conformant and handled externally.
• The pre-defined entities “train”, “signal” and “rail” refer to the current train, the next signal in direction of travel and the emitting xATC rail respectively
• “schedule()” simply schedules an event based on railway time.
• Actions (like doors(), speed(), ars()) are asynchronous in first term. You can make them explicitly synchronous by calling wait() without arguments on them. This only works on some actions (e.g. not on ars(), because that might never return)
• Alternatively, calling wait(“Eventname”) emits the specified event when the action is complete.

I imagine that the asynchronous code execution (wait etc.) can draw much inspiration from python's asyncio. Possibly we will also use an “await” keyword to wait for asynchronous tasks. https://docs.python.org/3/library/asyncio-task.html

The above code using an “await” keyword:

On Approach:
train.text_outside = "E3 - Trisiston\nvia Tanh Cliffs"
train.text_inside = "Next Stop: Euler Street\nTerminus, please get off."
train.ars = false
local stop_await = train:stop_at_rail()
-- makes train stop at the rail, returns an 'awaitable'
stop_await:then("Stop")
-- when stopping the train is finished, throw the "Stop" event.
rail.digiline_send("DFI-controller", "Arr:E3W")
-- scope of "local stop_await" ends here

On Stop: -- user-defined event (fired by stop_at_rail() above)
train.reverse()
train.text_inside = "Euler Street"
train.doors("R")
schedule("+02;00", "Depart-Ready") -- Schedule departure in 2 minutes
rail.digiline_send("DFI-controller", "Stp:E3W")

On Depart-Ready:
signal:ars():then("Depart"):else("ARSFail") -- Tell the next signal to do ARS and emit "Depart" when route is set

On Depart:
await train.doors("C") -- Wait in-place (without another event) for doors to close
train.text_inside = "Welcome on board! Next stop: Mountain South"
train.ars = true
train.speed()
rail.digiline_send("DFI-controller", "Dep:E3W")

On ARSFail:
schedule("+00;10", "Depart-Ready") -- try again later

All awaitables can either complete successfully or fail. If an awaitable succeeds, and then(“eventname”) had been called on it, the specified event is fired. If the await keyword has been used, execution continues. However, if an awaitable fails (this can happen e.g. for the “stop_at_rail” event if a player takes over control during stopping, or for “ars” if the route is cancelled manually by a player), the coroutine that awaits it is cancelled. It is possible to call awaitable.else(“eventname”) to catch the error case in a separate event, however.

then() and else() are chainable, that means they return the awaitable they've been called on.

The code written in xATC is untrusted, since it is possibly written by untrusted users. We therefore build a somewhat custom Lua interpreter.

How this code is parsed and executed is not yet decided. This is a matter of where to lay the boundary between custom parsing and Lua compiling.

A first suggestion looks as follows:

• Every line is considered an instruction or a block start/end (we won't support ; to separate multiple commands per line for now)
• The global instruction structure is parsed by custom code
• Individual instructions are left as-is. Custom code assembles the instructions into Lua code safe for procesing.
• Instructions are only allowed to be in “Assignment” or “Function Call” format. It should not be possible to construct things like '(function() evil_stuff end)()'

Perhaps the exact syntax and semantics of “wait” functions will be changed in this draft, this is yet unclear. There needs to be a way to continue a halted “thread” from saved data, which effectively prevents us from using coroutines. The simplest way would be to abandon wait() altogether, but this violates the “simplicity” constraint.

A possible 'wait_for(“Event”)' function could allow for quasi-procedural operation, doing things like

On Approach:
schedule("/01;00+00;00", "Depart") -- When the train approaches, emit a "Depart" event at the next full minute
train.stop() -- Stop the train

On Stop:
wait_for("Depart") -- When the train stopped, wait until the "Depart" event arrived (if it already arrived before, returns immediately; this works here because there's no "On Depart" that handled this, it remained in queue)
train.speed()

Eventually, a further extension can implement event sending from external entities, such as a train waiting for another connecting train.

-- Train 1 (running in context of some xATC rail called "Trisiston:1")
On Stop:
wait_for("connecting_train_arrived")
schedule("+00;20", "Depart")

-- Train 2
On Stop:
rail.dispatch_event_to("Trisiston:1", "connecting_train_arrived")
-- the semantics of this are not yet specified