The thesis deals with security hazards in distributed environments where traditional centralized approaches are only of limited serviceability. One of the very successful model for treating security and access management in distributed systems are so called reputation systems. The main goal of the rep- utation systems is to provide entities in the environment with mechanisms for inferring and building trust consequently used for access control. If the trust between two entities is high enough, transactions are likely to be allowed. The thesis proposes a new security model with trust management system for dynamic and distributed environments with huge number of entities. In dynamic systems new entities or relationships are likely to emerge or existing entities or relationships may often disappear. Such dynamics pose severe problems even for traditional reputation systems. Therefore our approach differs from the traditional ones in the way adopted for establishment and management of trust between entities in our point of view trust is not assigned to particular relationships but the trust is common for a group of entities. In this way, our proposal significantly enhances ability to infer trust between entities with no previous personal experiences with each other or in environments with huge number of entities. For the proposal differs in understanding of trust, it uses a hypergraph model for representation of system of entities. The security model proposed in the thesis contains two algorithms for transformation of a general input graph structure into hypergraph model, an algorithm treating dynamics of the distributed environment and a security subsystem. Our experimental implementation SecGrid utilizes proposed algorithms and it is used for experimental verification of the security models. The experiments investigate ability of the transformation algorithms; in details the dynamic part of our proposal together with the security subsystem proposed specially for the hypergraph model. Experiments show that our model overcame the traditional graph model in many ways especially in dynamic environments with huge amount of entities.