如何在 Detectron 2 上使用自定义（或蛋白）增强？

Question

我有下面的代码，我在

Detectron 2

训练期间使用默认增强，但问题是对我有用的增强很少。

1. 这是我用自己的函数要实现的目标。

2. 这个github问题链接试图解决。

想知道这是否是正确的方法，而且，如果我想看看发生了什么，我该如何查看增强图像结果？

import detectron2.data.transforms as T
from detectron2.data import detection_utils as utils

def custom_mapper(dataset_dict):
    
    dataset_dict = copy.deepcopy(dataset_dict)
    image = utils.read_image(dataset_dict["file_name"], format="BGR")
    transform_list = [T.RandomBrightness(0.8, 1.2),
                      T.RandomContrast(0.8, 1.2),
                      T.RandomSaturation(0.8, 1.2),
                      ]
    image, transforms = T.apply_transform_gens(transform_list, image)
    dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))

    annos = [
        utils.transform_instance_annotations(obj, transforms, image.shape[:2])
        for obj in dataset_dict.pop("annotations")
        if obj.get("iscrowd", 0) == 0
    ]
    instances = utils.annotations_to_instances(annos, image.shape[:2])
    dataset_dict["instances"] = utils.filter_empty_instances(instances)
    return dataset_dict

但问题是

1. 我想构建我的自定义增强功能或使用 albumentations 来实现此目的。
2. 我不想每次都使用所有的增强，并且我使用的上述转换中没有可能性。因此，一种解决方法是对个人或团体使用类似

   OneOf

   的内容。

Answer 1

因此，在运行完代码流程和文档后，我发现每个

Augmentation

类都依赖于从

fvcore库继承的

Transform类

同样的依赖关系也被模糊地定义了在此突出显示的文档块中

然后查看上述所有方面，然后浏览这个 github PR 合并战斗线程，我感觉我走在正确的轨道上，因此在

PILColorTransform

增强之后，我编写了这段代码，该代码非常适合自定义增强

1。定制增强功能

class GenericWrapperTransform(Transform):
    """
    Generic wrapper for any transform (for color transform only. You can give functionality to apply_coods, apply_segmentation too)
    """

    def __init__(self, custom_function:Callable):
        """
        Args:
            custom_function (Callable): operation to be applied to the image which takes in an ndarray and returns an ndarray.
        """
        if not callable(custom_function):
            raise ValueError("'custom_function' should be callable")
        
        super().__init__()
        self._set_attributes(locals())

    def apply_image(self, img):
        '''
        apply transformation to image array based on the `custom_function`
        '''
        return self.custom_function(img)

    def apply_coords(self, coords):
        '''
        Apply transformations to Bounding Box Coordinates. Currently is won't do anything but we can change this based on our use case
        '''
        return coords

    def inverse(self):
        return NoOpTransform()

    def apply_segmentation(self, segmentation):
        '''
        Apply transformations to segmentation. currently is won't do anything but we can change this based on our use case
        '''
        return segmentation


class CustomAug(Augmentation):
    """
    Given a probability and a custom function, return a GenericWrapperTransform object whose `apply_image`  will be called to perform augmentation
    """

    def __init__(self, custom_function, prob=1.0):
        """
        Args:
            custom_op: Operation to use. Must be a function takes an ndarray and returns an ndarray
            prob (float): probability of applying the function
        """
        super().__init__()
        self._init(locals())

    def get_transform(self, image):
        '''
        Based on probability, choose whether you want to apply the given function or not
        '''
        do = self._rand_range() < self.prob
        if do:
            return GenericWrapperTransform(self.custom_function)
        else:
            return NoOpTransform() # it returns a Transform which just returns the original Image array only


def white(image):
    return np.ones(image.shape, dtype = np.uint8)*255 # returns white Image

def black(image):
    return np.zeros(image.shape, dtype=np.uint8) # returns black image

def rand(image):
    return np.random.randint(0,256,image.shape, dtype = np.uint8) # returns random image

def default(image):
    return image # returns original image

2。实现类似函数

OneOf

我研究了AugmentationList

的实现并构建了自己的代码来随机生成转换。设置

k=1

模仿

OneOf

的功能。

class KRandomAugmentationList(Augmentation):
    """
    Select and Apply "K" augmentations in "RANDOM" order with "Every"  __call__ method invoke
    """
    def __init__(self, augs, k:int = -1):
        """
        Args:
            augs: list of [Augmentation or Transform]
            k: Number of augment to use from the given list in range [1,len_augs]. If None, use all. If it is -1, generate K randomly between [1,len_augs]
        """
        super().__init__()
        self.max_range = len(augs)
        self.k = k
        self.augs = augs # set augs to use as fixed if we have to use same augs everytime
    

    def _setup_augs(self, augs, k:int):
        '''
        Setup the argument list. Generates the list of argument to use from the given list
        args:
            augs: list of [Augmentation or Transform])
            k: Number of augment to use from the given list in range [1,len_augs]. If False, use all. If it is -1, generate K randomly between [1,len_augs]
        '''
        if k == -1: # Generate a random number
            k = np.random.randint(1,len(augs)+1)
        
        elif k is None: # use all
            k = self.max_range

        temp = np.random.choice(augs,k,replace=False) # get k augments randomly
        return [_transform_to_aug(x) for x in temp]

    
    def __call__(self, aug_input) -> Transform:
        tfms = []

        for x in self._setup_augs(self.augs, self.k): # generate auguments to use randomly on the fly
            print(x)
            tfm = x(aug_input)
            tfms.append(tfm)
        return TransformList(tfms)

    def __repr__(self):
        msgs = [str(x) for x in self.augs]
        return "AugmentationList[{}]".format(", ".join(msgs))

    __str__ = __repr__

3.将所有东西放在一起

from detectron2.data import transforms as T
import numpy as np
from PIL import Image

augs = KRandomAugmentationList(
        [
        # my custom augs
        CustomAug(white), 
        CustomAug(black),
        CustomAug(default),
        CustomAug(rand),
        
        # augs from Detectron
        T.RandomBrightness(0.4, 1.6),
        T.RandomSaturation(0.4, 1.6),
        T.RandomContrast(0.4,1.6),
        T.RandomCrop("absolute", (640, 640)),
        T.RandomFlip(prob=0.5),
        ],
        k = -1)



# Calling the below block multiple times will give you different combinations
# of Augmentations everytime depending on the value of `k`

image = np.array(Image.open("my_image.png")) # RGB image array
input_ = T.AugInput(image, boxes=None, sem_seg = None) # boxes and segments are optional
transform = augs(input_)  # Apply the augmentation
image_transformed = input_.image  # augmented image

Image.fromarray(image_transformed) # show RGB image

---

Answer 2

感谢您分享您的见解！ 我有一个问题：我尝试使用一些来自 Albumentations 的增强功能。

但是我收到一个错误，预期有 1 个参数，但给出了 2 个参数。您还记得关于此的某事或类似的事吗？

def gaussianNoise(image): 
    return albumentations.GaussNoise(var_limit=(10.0, 50.0), p=0.5(image=image)

我尝试这样称呼它：

augmentation=[
                L(T.RandomFlip)(),
                L(T.ResizeShortestEdge)(
                    short_edge_length=short_edge_length,
                    max_size=max_size,
                    sample_style="choice",
                ),
                L(T.RandomBrightness)(
                    intensity_min=0.8,
                    intensity_max=1.2,
                ),
                L(T.RandomContrast)(
                    intensity_min=0.8,
                    intensity_max=1.2,
                ),
                L(A.CustomAug)(
                    A.gaussianNoise, prob=0.5
                 )
            ],

我必须实现一个 call() 以及这个函数需要什么参数吗？

Answer 3

您可以使用

albumentations

轻松完成这两件事。

这是一个例子：

import albumentations as A

def create_transform():
    return A.Compose(
        [
            A.OneOf([
                A.RandomBrightness(p=1),
                A.RandomContrast(limit=.2, p=1)
                ],
                p=.8),
            A.GaussNoise(p=.8),
        ],
    )

def transform_image(image):
    
    transform = create_transform()
    
    transformed = transform(image=image)
    
    transformed_image = transformed["image"]
    
    return transformed_image 

对于每个蛋白操作都有一个概率参数

p=1

，对于在两个或多个之间随机选择一个操作，您可以使用

A.OneOf([],p=.8)

。在我的示例中，

A.OneOf([],p=.8)

和

A.GaussNoise(p=.8)

都有 80% 的机会应用于图像。

使用随机图像进行测试

import numpy as np

image = np.random.randint(255, size=(24,24)).astype("float32")

# array([[104.,  29., 230.,  68., 134.],
#        [ 43., 162.,  73., 246., 111.],
#        [ 71., 188., 206.,  85., 233.],
#        [105., 178., 190.,  54., 231.],
#        [  0., 116., 169., 244., 178.]], dtype=float32)

transform_image(a)

# array([[1.        , 1.        , 1.        , 1.        , 0.11930324],
#        [1.        , 1.        , 0.        , 0.        , 1.        ],
#        [0.        , 0.        , 1.        , 0.5334698 , 0.        ],
#        [1.        , 0.        , 0.        , 1.        , 1.        ],
#        [0.        , 0.        , 1.        , 0.        , 1.        ]],
#       dtype=float32)