Targets are the way used by the pipeline to know if the expected Inputs/Outputs (I/O) of a task are existing/produced. A target should contain all the information necessary to the pipeline to make its decision on running or not the next step of the chain of treatment.
An instance of a target is fully identified by the its identifier, its version and the name of the file. Thus, the target can be seen as a category/group of file with a physical representation of them.
A target instance can be created simply by instantiating a
poppy.pop.target.Target
class. The mandatory parameters have to be
given at the instantiation. For example, a target for the dataset
ROC-SGSE_LZ_MEB-SGSE-TEST-LOG
in version 01
, and a physical
file /path/to/dataset/file
:
from poppy.pop import Target
target = Target(
"/path/to/dataset/file", # filename of the dataset
"ROC-SGSE_LZ_MEB-SGSE-TEST-LOG", # identifier for the target
"01", # version of the target
)
This results in a target instance that can be used and shared across the pipeline, through the context. Placing it in the context allows the pipeline to easily find targets used as I/O for tasks. See target_usage for details.
A target can also be created from a task instance created from a plugin. The plugin contains all necessary information for the target creation, allowing an easy and maintainable way of referencing and creating target instances. See Tasks.
If for example the target is named xml_test_log
in the descriptor file
of a plugin called DARE-SGSE
, a target instance can easily be created
with (assuming a task named to_xml_file
):
from poppy.pop.plugins import Plugin
# create the class for the task
Task = Plugin.manager["DARE-SGSE"].task("to_xml_file")
# create a fake functionality for the task
@Task.as_task
def run(task):
"""
task is an instance of the task to_xml_file, created from a plugin. It
can be used to create a target from informations provided in the
descriptor.
"""
# first get the class of the target
Target = task.target("xml_test_log")
# create the instance with a path to the file
target = Target("/path/to/target")
Danger
If it is an input target, input_target
must be used instead of
target
as method on the task instance.
All status changes of a target must be done inside its context. It allows the target to intercept any problem occurring while generating a file in input or output, and to gives the good error information to the POPPy framework.
For example, to use the context of the target and open the file inside it:
# use the context of the target
with target.activate():
# open the file
with target.open("r") as f:
# do things with the file
This example is simple, but shows how the target handles all the job for the
user. The status of the target is automatically updated according to each step.
In case of an error inside this context, the status of the target is set
accordingly to poppy.pop.target.Target.error()
and reported into the
database.
At startup, it is marked as poppy.pop.target.Target.ok()
and
poppy.pop.target.Target.pending()
.
If while treating the file, the target is detected as empty, the :exception:`poppy.pop.target.Target.TargetEmpty` must be raised, handled by the context to mark the target as empty. It is used by the pipeline to know that a file has been treated correctly, but that it is not existing, and this is normal.
The available status of a target are listed in API.