With the advent of the Internet-of-Things way more sensor-generated data streams came available that researchers want to exploit context from. Many researchers worked on context recognition for rather unimodal data in pervasive systems, but recent works about object virtualisation in the Internet-of-Things domain enable context-exploitation based on processing multi-modal information collected from pervasive systems. Additionally to the sensed data there is formalised knowledge about the real world objects emitted by IoT services as contributed by the author in [1], [2] and [3]. In this work an approach for context recognition is proposed that takes knowledge about virtual objects and its relationships into account in order to improve context recognition. The approach will only recognise context that has been predefined manually beforehand, no new context information can be exploited with the work proposed here. This work’s scope is about recognising the activity that a user is most likely involved in by observing the evolving context of a user of a pervasive system. As an assumption for this work the activities have to be modelled as graphs in which the nodes are situations observable by a pervasive system. The pervasive system to be utilised has to be built compliant to the Architectural Reference Model for the IoT (ARM) to which the author has contributed to in [4] and [5]. The hybrid context model proposed in this thesis is made of an ontology-based part as well as a probability-based part. Ontologies assist in adapting the probability distributions for the Hidden Markov Model-based recognition technique according to the current context. It could be demonstrated in this work that the context-aware adaptation of the recognition model increased the detection rate of the activity recognition system.