A prefix-list filter was created to reject certain routes from OSPF protocol, but it did not work, and the route was still installed in routing table.  

The mentioned behavior is expected per design.

Configuring OSPF Routing Policy | Junos OS | Juniper Networks

According to Juniper OSPF Routing Policy guide, the import router filter applies to external routes ((LSAs type 5 and 7)) in OSPF only. An external route is a route that is outside the OSPF Autonomous System (AS).

The filter could filter the routes, but not link-state address (LSA) flooding. It does not impact the OSPF database, which means, the intra-area and interarea routes are not considered for filtering. The default action is to accept the route when the route does not match the policy.

If it is necessary to filter the unwanted intra-area and interarea routes, an export policy should be configured on the directly connected upstream/downstream router to prevent the distribution of the unwanted routes.